COMMENTS ON THE SYMBOL GROUNDING PROBLEMAndrzej J. Pindor, University of Toronto, Computers and Communications pindor@utirc.utoronto.ca .QP .B Abstract .R A solution to the symbol grounding problem proposed by Harnad requires giving a system both linguistic and sensorimotor capacity indistnguishable from those of a human. The symbols are then grounded by the fact that analog sensorimotor projections on transducer surfaces, coming from the real world objects, and successively formed sensory invariants of nonarbitrary shape constrain the symbol combinations over and above what is imposed by syntax, and tie the symbols to those real objects. It is argued here that the full sensorimotor capacity may indeed be a crucial factor, since it is capable of providing the symbols (corresponding to language terms) with a deep underlying structure, which creates a network of intricate correlations among them at a level of primitive symbols based in inputs from the transducers. On the other hand the nonarbitrary shapes of sensory invariants as well as the analog nature of sensorimotor projections seem to be of no consequence. Grounding is then seen as coming from this low-level correlation structure and, once known, could in principle be programmed into a system without a need of transducers. .RE In a series of papers Stevan Harnad has suggested a solution to the "symbol grounding" problem (Harnad 1990, Harnad 1993, Harnad 1993a). The essence of the problem is that symbols manipulated by digital computers or even neural nets (in SIM or IMP implementations, see Harnad 1993a) do not seem to be about anything in particular - their only meaning comes from the mind of an interpreter. The symbols themselves are manipulated according to syntactic rules, on the basis of their shapes only, these shapes being unrelated to what the symbols can be interpreted as standing for. This lack of meaning of the symbols is, Harnad claims, evident for instance from the fact that one cannot learn Chinese from a Chinese-Chinese dictionary (Harnad 1993a). Consequently, there is no guarantee that a TT-passing system, say in Chinese, really understands Chinese - it may be simply manipulating symbols (Chinese characters) syntactically, without any regard for what these symbols are _about_. Harnad suggests that symbols of a system can be grounded if (and only if) the system can pass a Total Turing Test, i.e. have both linguistic _and_ sensorimotor capacity totally indistinguishable from our own. Such a system would have to be equipped with a full range of transducers giving it a complete, as he calls it, robotic capacity. Harnad then proposes a more detailed model describing how the robotic capacity leads to the grounding of symbols. He argues that inputs from senses (or sensors for a robot) in the form of analog "sensory projections" connect to symbols of the system (i.e. language terms) through sensory invariants of _nonarbitrary shape_ (these invariants, he suggests, could be extracted from the sensory projections, in the case of a robot, using neural nets). This fact, according to him, puts additional constraints on symbol combinations, over and above syntactic constraints, and results in grounding. Symbols are about real world objects whose sensory projections invariants they correspond to (Harnad 1993a). Before discussing Harnad's suggested solution to the symbol grounding problem it may be appropriate to comment on his use of the term "symbol". In most of his arguments this word is used in the meaning of a computational token capable of being interpreted as being _about something_, i.e. corresponding to a language term. However he also talks about "...the manipulations of physical 'symbol tokens' on the basis of syntactic rules that operate only on the 'shapes' of the symbols (...), as in a digital computer or its idealization, a Turing machine manipulating, say, 0's and 1's " (Harnad 1993a). This indicates that he considers a digital computer 0's and 1's also as (perhaps primitive) symbols, which the higher level symbols, capable of being interpreted as corresponding to language terms, are built of. This is an important point, relevant to my discussion below of Harnad's stress on the analog nature of 'sensory projections'. The main idea of Harnad's model, the need for a system to have the full sensorimotor capability of a human being in order for its symbols to be grounded, expresses the fact that the terms of a language we use are not defined by their relationships to (correlations with) other language terms only - they are defined by a cooperation, so to speak, of all sensory inputs at our disposal. When we say, for instance, "cat" , understanding of this term involves all experiences we have had with cats - through vision, touch, smell etc. A single language dictionary (like Chinese-Chinese in Harnad's example) can only relate language terms among themselves. Relating them to real world objects requires full sensorimotor capacity indistinguishable from our own (Harnad 1993). No surprise that a TT-passing system, which demonstrably does not have such a capacity (for instance Searle's Chinese Room, see Searle 1980) can be suspect with respect to its understanding of the language it seems so expertly to use. After all we do not expect a person blind from birth to understand how colours influence our interaction with the world, regardless of the amount of verbal explanations. How the sensorimotor inputs lead to grounding of top-level symbols (i.e. language terms) is another story and below I criticize two aspects of the Harnad's model. The first aspect which I would like to discuss is his claim of "nonarbitrary shapes" of sensory invariants extracted from sensory projections from real world objects onto a system's transducer surfaces. The word 'shape" above has to be interpreted in a somewhat generic sense - in the case of senses other than vision and touch it must mean some particular feature of the sensory invariants which is somehow fixed by the nature of an object (phenomenon) it corresponds to. This "nonarbitrariness of shape", in Harnad's eyes, imposes a constraint on a system's symbols assigned to represent such an invariant. To what extent are the shapes of sensorimotor projections 'nonarbitrary'? I will consider below several examples indicating that the shapes of the sensorimotor projections seem to be to a large extend dependent on the physical nature of the transducers, which are, in a sense, results of evolutionary 'accidents' (naturally optimized within an accessible range of physical parameters) and are then to a large degree arbitrary. 1. Colours. Colour vision is dependent on six types of cells in the eye's retina sensitive to light in various parts of the spectrum (DeValois, Abramov and Jacobs 1966, DeValois and Jacobs 1968). Two of these have to do with perception of blue and yellow, two with perception of red and green and two are sensitive only to intensity of light within the 'visible range'. Terms 'blue', 'yellow', 'green' and 'red' refer to various ranges of light wavelength covering the visible portion of the spectrum. Now, it is most likely an evolutionary 'accident' how the visible spectrum is divided into these four regions. With a somewhat different chemistry the ranges of sensitivity of the colour cells might have been different, resulting in a different colour perception. One can also conceivably imagine that, had the evolution of the human eye gone somewhat differently, we might have ended up with a colour vision mechanism distinguishing three or five colours. Consequently, sensory projections of real objects, coming from the colour vision system, would have different "colour shapes", which are to a large extent determined by the physical nature of the _transducers_ and not the objects themselves. 2. Visual shapes. Due to the nature of human eye optics, projections of real objects on the eye's retina are already distorted - for instance many straight lines in the outside world project on the retina as curved lines. In addition, as is well known, these projections are upside down. The fact that we see the real world objects "right way up" is a result of the brain learning to _correlate_ shapes of sensory projections from the visual system with other sensory projections. If we perform an experiment in which subjects are made to wear glasses inverting the image falling on the retina upside down (so that it now correspond to the "real" situation), the subjects are at first very confused and have difficulties moving around, grabbing objects etc. However, after a certain time there seems to be a discontinuous transition to a state in which the subjects report that they see everything "normally" and have no more problems performing tasks requiring vision. Obviously, their brains have learned to _correlate_ the new "shapes" of the sensory projections from the vision system with other sensorimotor projections. A similar effect arises if we try to trace a pattern (say with a stylus) looking not at the pattern itself, but at its reflection in a mirror. Initially we are quite confused, but if we persist at the task after a while it becomes as natural as tracing the original pattern - the brain learns to compensate for the reversal of left and right. One could also speculate that if in the distant past evolution chose a slightly different route we might have ended up with eyes more like those of insects - sensory projections of our visual system, coming from real world objects, would be very different and there is no reason to doubt that our brain would learn to deal with such a situation. We see again that the shapes of the sensory projections are in some sense arbitrary, determined by the physical nature of the transducers. 3. Touch. Let us perform a very simple experiment - we cross the index finger with the middle finger of our right hand in such a way that the tip of the middle finger is to the left of the tip of the index finger (and vice versa). Now if we touch a small round object with these two fingers simultaneously (i.e. the object touches the left side of the tip of the index finger and the right side of the tip of the middle finger) we have an impression that we are touching two objects and not one. We see that even such basic information about real objects as whether we deal with a single object or with two separate objects cannot be reliably extracted from a single sensory projection - we need _correlations_ from other sensory projections to form a picture which makes sense. The above examples seem to put in doubt Harnad's claim that "nonarbitrary shapes" of sensorimotor projections from real objects onto transducer surfaces are a crucial element of symbol grounding. The shapes of the sensorimotor projections are shown to be arbitrary to a large extent and it is the _correlations_ among these projections which appear to play a dominant role. Harnad illustrates categorization process leading to the grounding of the category 'names" by an imaginary example of learning to distinguish between edible and poisonous mushrooms (Harnad 1993). It is interesting to note that in his example the grounding of the mushroom names ("mushrooms" for the edible ones and "toadstools" for the poisonous ones) takes place on the basis of _correlations_ between various sensory projections. _Shapes_ of the projection invariants do not to enter in any way. The second aspect of Harnad's model is his claim that the sensorimotor projections coming from the system's transducers, fed subsequently to a neural net for the purpose of categorisation (extraction of invariants), are analog. For instance he writes (Harnad 1993): "...it [Harnad's model] is 3-way (analog-connectionist-symbolic) with the connectionist component just a place-holder for any mechanism able to learn invariants in the analog sensorimotor projections that allow the system to do categorisation" and further down: "...performance requirements of such a T3 [i.e. TTT] -scale robot depend essentially on analog and other nonsymbolic forms of internal structure and function." However nowhere in his arguments does Harnad convincingly show that this analog feature of the input (in the form of sensorimotor projections) to neural nets which do the invariant extraction is, in fact, essential. Any analog signal can be approximated with arbitrary accuracy by a digital signal. Since neural nets can have only finite sensitivity, whether they are fed an analog signal or a correspondingly finely graded digitized signal cannot matter for further processing. Once we accept this, these digitized signals from the transducers (sensorimotor projections) can be viewed as primitive symbols, in the same spirit as 0's and 1's of a Turing machine. All further processing can be considered as symbol manipulations which one way or another lead to construction of high-level symbols representing language terms (category names). This may very well happen with the use of neural nets to extract invariants from sensory projections and perhaps perform categorization. Since any neural net may be emulated using a suitably programmed digital computer, all these steps can be achieved without a need for analog devices. The above analysis suggests that full robotic capacity of a system might provide high-level symbols with a deeper structure based in correlations among the primitive symbols, the sources of which are inputs from sensorimotor transducers. Symbol grounding would then be achieved by the presence of such an underlying structure, which would give the symbols a much richer (and more intricate) set of relationships than can be offered by a (single-language) dictionary. These relationships mirror the experiences of interacting with the real world, making the symbols effective in such an interactions and justifying the claim that the symbols are grounded. It is nevertheless worth pointing out that there does not seem to be a reason why the underlying structure discussed above, once established, could not be built (programmed) into a symbolic system, without a need to give the system the full robotic capacity. Such a system would be capable of passing the TT and should perhaps also be considered to posses understanding of the language it uses. There is one more aspect of the grounding problem, as discussed above, which requires mentioning. There are situations when we deal with concepts defined solely using language, without a reference to sensorimotor projections from real world objects. Such situations arise, for instance, in the case of mathematics. If we consider abstract set theory or abstract group theory, we define objects (sets, group elements) purely syntactically and then proceed to draw all possible conclusions concerning the consequences of these definitions. In spite of the fact that the symbols we manipulate do not require grounding in sensorimotor projections from real world objects, and the manipulations depend only on shapes of these symbols (which are completely arbitrary), we do talk about "understanding" mathematics (abstract set theory, abstract group theory, etc.). It is clear that understanding in this case means a knowledge of (or ability to deduce) _correlations_ among symbols of increasing complexity, arising from definitions of basic symbols from which these higher level symbols are constructed. In conclusion, it is argued above that even though two aspects of Harnad's model for symbol grounding seem unjustified: - shapes of sensorimotor projections from real objects onto transducer surfaces do not appear to be relevant and hence cannot play a role in restricting symbol combinations; - importance of the analog nature of the sensorimotor projections, fed subsequently to neural nets for invariant feature extraction, is not apparent (there are reasons to think that these projections might just as well be digitized leaving us with pure symbol manipulations); the main idea of the model - TTT capacity - may be crucial for symbol grounding. It may be the combinations of various sensorimotor experiences with real objects which lead to the formation of a deep structure underlying the high-level symbols which provides (epistemological) meaning of language terms. This structure underlying the symbols may be somewhat akin to to the semantic structure of language J. Katz is attempting to establish in "The Metaphysics of Meaning" (Katz 1990), although he takes a definitely platonic view, whereas the structure referred to here has a very specific sensorimotor basis. There also appears a possibility that if a symbolic system works on the basis of digitized inputs, corresponding to sensorimotor projections coming from transducers, as basic symbols, it might posses understanding without TTT capability. Possibility of ascribing understanding to a purely symbolic system seems in accordance with using the term "understanding" in the case of abstract mathematics, where (mathematical) terms used are described verbally only, without recourse to the full sensorimotor capacities of a human being. .B References .R DeValois, R.L., I. Abramov, and G.H. Jacobs, (1966) Analysis of Response Patterns of LGN Cells. Journal of the Optical Society of America 56:966-77. DeValois, R.L., and G.H. Jacobs, (1968) Primate Color Vision. Science 162:533-40. Harnad, S. (1990) The Symbol Grounding Problem. Physica D 42: 335-346. Harnad, S. (1993) Symbol Grounding is an Empirical problem: Neural Nets are Just a Candidate Component. Proceedings of the Fifteenth Annual Meeting of the Cognitive Society. NJ: Erlbaum Harnad, S. (1993a) Grounding Symbols in the Analog World with Neural Nets. Think 2:12-78 (Special Issue on "Connectionism versus Symbolism" D.M.W Powers & P.A. Flach, eds.). Katz, J.J. (1990) The Metaphysics of Meaning, MIT Press, Cambridge, Massachusetts. Searle, J. R. (1980) Minds, brains and programs. Behavioral and Brain Sciences 3: 417-424. |