The New Connectionism: Developing
Relationships Between Neuroscience and
Artificial Intelligence
by Jacob T. Schwartz
Part of the confidence with which artificial intelligence researchers view the prospects of their field stems from the materialist assumptions they make. One is that "mind" is simply a name for the information-processing activity of the brain. Another is that the brain is a physical entity that acts according to the laws of biochemistry and is not influenced by any irreducible "soul" or other unitary, purely mental entity that is incapable of analysis as a causal sequence of elementary biochemical events. This broadly accepted view, together with the rapidly mounting mass of information concerning nervous system physiology, microanatomy, and signaling behavior and with the current technology-based push to construct analogous computing systems involving thousands of elements acting in parallel, has encouraged a shift in emphasis among AI researchers that has come to be identified as "the new connectionism." The emphases that characterize this school of thought are as follows:
1. The brain operates not as a serial computer of conventional type but in enormously parallel fashion. The parallel functioning of hundreds of thousands or millions of neurons in the brain's subtle information-extraction processes attains speed. Coherent percepts are formed in times that exceed the elementary reaction times of single neurons by little more than a factor of ten. Especially for basic perceptual processes like sight, this observation rules out iterative forms of information processing that would have to scan incoming data serially or pass it through many intermediate processing stages. Since extensive serial symbolic search operations of this type do not seem to characterize the functioning of the senses, the assumption (typical for much of the AI-inspired cognitive science speculation of the 1960-80 period) that serial search underlies various higher cognitive functions becomes suspect.
2. Within the brain, knowledge is stored not in any form resembling a conventional computer program but structurally, as distributed patterns of excitatory and inhibitory synaptic strengths whose relative sizes determine the flow of neural responses that constitutes perception and thought.
AI researchers developing these views have been drawn to involvement in neuroscience by the hope of being able to contribute theoretical insights that could give meaning to the rapidly growing, but still bewildering, mass of empirical data being gathered by experimental neuroscientists (many of whom regard theoretical speculation with more than a little disdain). These AI researchers hope to combine clues drawn from experiment with the computer scientists' practiced ability to analyze complex external functions into patterns of elementary actions. By assuming some general form for the computational activities characteristic of these actions, they hope to guess something illuminating about the way in which the perceptual and cognitive workings of the brain arise. That is to say, computer scientists hope to relate to experimental neuroscience much as theoretical physicists relate to experimental physics-by contributing unifying theoretical insights and theoretically based conjectures that can guide experimentation along fruitful paths.
The awesome complexity of the brain poses major obstacles to easy realization of this aim. The magnitude of the problems that need to be unraveled is indicated by a few intimidating estimates and a brief review of some basic facts of neuroscience. The human brain consists of approximately 100 billion neurons, possibly even ten times as many. Neurons usually communicate by transmitting discrete electrical spikes (action potentials) to a population of follower neurons. As far as is known, the precise amplitude and shape of such a spike, and the precise time of its arrival within an interval of two milliseconds or so, are physical details that the nervous system is not able to exploit. Hence, one can model each spike as a single information-carrying "bit" in a neuron's output stream and say that a neuron outputs information at a rate of approximately 100 bits per second. This way of thinking leads to an estimate of 10 trillion bits per second, give or take a factor of 100, for the internal "bandwidth" of the brain.
The computational activity of each neuron involves a great variety of mechanisms, still most imperfectly understood. Nevertheless, a considerable mass of experimental evidence supports the following general picture. A neuron transmits information to its follower neurons at interneuron junctions called synapses. A single neuron can have as many as 10 thousand synaptic inputs, though in some cases many fewer inputs, and in other cases as many as 100 thousand inputs, converge on single neurons. The total number of synapses in the brain can be estimated as 1,000 trillion, though this estimate, like all those offered in the next few paragraphs, is uncertain by a factor of roughly 100. Input signals transmitted to a neuron (generally chemically) across a synapse trigger a wide variety of reactions. One is modulation of the ionic conductivity of the affected neuron's membrane, which either raises the voltage of a portion of its interior (excites the neuron) or lowers this voltage (inhibits the neuron). After attenuation in space and time in a manner determined by the chemistry and geometry of the affected neuron and its synapses, the neuron then combines the voltage changes generated by such synaptic effects. If the resulting combined (summed, for example) voltage exceeds a reaction threshold, the neuron generates an output spike or other electrical signal. This is then transmitted to all its output synapses. Though many other mechanisms play a role, this kind of effect seems basic to many of the fastest computations performed by the brain.
Other forms of synaptic input are known to have slower but longer-lasting biochemical effects than these ionic effects, which probably support the bulk of the brain's information-transmuting activity. Stimulation of certain synapses can, for example, trigger enzymatic activities within a neuron that modify its biosynthetic activities-for instance, by increasing or decreasing its susceptibility to excitatory or inhibitory stimuli that are acting ionically. Depending on the chemical effects involved, such synaptic modification of fast ionic responses may last for as little as fifty milliseconds or for as long as several seconds, minutes, or days; it may even become permanent. Other synaptically triggered enzymatic reactions can initiate sequenced biochemical changes. For example, a neuron's electrical response may be enhanced for several tens of milliseconds but then inhibited for a longer period, leading to complex patterns of alternation between excitation and inhibition. The variety of single neuron behaviors that the wide spectrum of enzymatic actions can engender has been explored in simple animals such as Aplysia, some of whose neurons are known to have highly individualized patterns of continuing, periodic, or burst activity.
Though it is not easy to summarize the wide range of synaptic response patterns with a few numbers representing the information processing power and storage capacity of a single neuron, the following estimates do not seem unfair. One byte (eight bits, about one printed character) may well suffice for representing the long-term strength of each synapse. Four additional bytes can then be taken to give a sufficiently complete representation of the short-term biochemical state of both sides of a synapse and of the state of the corresponding synaptic gap, as determined by its stimulation history up to a given moment. Such exceedingly rough quantitative guesses lead us to estimate that the long-term memory available to the brain is about 10,000 trillion bytes and that the amount of shorter-term data needed to characterize the state of each of its synapses is roughly the same. The logical activity of each neuron can then be regarded as a process that combines approximately 10 thousand input bytes with roughly 40 thousand synapse status bytes at a rate of 100 times each second. The amount of analog arithmetic required for this estimate is (again very roughly) 10 million elementary operations per neuron per second, suggesting that the computing rate needed to emulate the entire brain on a neuron-by-neuron basis may be as high as 1,000,000 trillion arithmetic operations per second. (Of course, computation rates many orders of magnitude lower might suffice to represent the logical content of the brain's activity if it could be discovered what this is.)
It is interesting to compare these exceedingly coarse estimates with corresponding figures for the largest supercomputer systems likely to be developed over the next decade. These will probably not attain speeds in excess of 1 trillion arithmetic operations per second, which is about one one-millionth of the computation rate that we have estimated for the brain. Today's large magnetic storage disks hold around 1 billion bytes of digital information each, which is roughly one ten-millionth of the storage capacity that we have ascribed to the brain. Even if we assume continuing rapid advances in storage technology and systems equipped with hundreds of storage disks, supercomputers seem unlikely to achieve more than 1 percent of the brain's storage capacity over the next decade. Clearly, the neuroscientist confronts a system whose workings are difficult to approach physically and whose operations are of awesome complexity.
CLUES TO BRAIN FUNCTION
One of the most salient clues from which the "connectionist" theorists hope to work is the observation that mental (especially sensory) processes seem to be of very restricted "depth," in the sense that not many successive elementary neural reactions are required to form the higher-level reactions that the brain generates. There is simply not time for very many successive reactions to be involved.
This is only a weak clue, however. Since neurons are vastly more complex than the elementary switches used to construct computers, a single stage of neural processing may compare to ten or more stages of electronic processing by elementary switchlike elements. Hence, outputs that the brain can generate in one-tenth of a second may compare in complexity with outputs requiring a hundred or more stages of processing by electronic switches. Moreover, so little is understood concerning the logical significance of the interconnections in the nervous system (even in cases where we know a great deal about the microanatomical structures involved) that it is hard to rule out any one of hundreds of conjectures about the way in which electronic devices should be connected so as to imitate the workings of the brain. A computer scientist, given a vast, almost totally unknown computer like the brain, with trillions of active elements connected in unfathomed ways, and asked to guess its mode of functioning with no other clue than the statement that it generates its outputs using highly parallel computations involving only a few hundred serially successive stages of processing, could feel only the most minimal confidence in whatever guess he or she ventured. The problem is not that we cannot imagine how known properties of neurons could serve to support intelligent function; it is that too many lines of speculation lie open for definitive choice between them to be feasible without additional evidence. The theorist's task is therefore to cultivate a sensitivity to the clues that are available in the enormous and growing, but confusing, mass of data that laboratory neuroscience already furnishes. In the paragraphs that follow I review a few of the most helpful clues.
Direct recording of the activity of single neurons has been possible for several decades, and by correlating controlled sensory inputs with single-neuron recordings one can get a crude picture of the workings of the brain's sensory systems, at least for the initial stages of neural processing. These stages seem to prepare incoming data for the first (still entirely mysterious) acts of recognition. Studies of this type suggest that certain general structures are common to several sensory modalities. In many cases, neurons that handle information generated by primary sensory systems having a natural one- or two-dimensional layout seem to be arranged in successive two-dimensional sheets (either within the cerebral cortex or in various smaller brain structures underneath the cortical lobes). The arrangement of cells in these sheets often seems to reflect the natural geometry, or at any rate some informationally significant dimension, of the sensory data itself. For example, the cells that accomplish the very first stages of image processing in the visual cortex are arranged retinotopically, which is to say in relatively precise one-to-one continuous correspondence with the retina of the eye (or, equivalently, with the geometry of images falling onto the retina). Cells devoted to the analysis of tactile sensations detected in the skin are arranged in much coarser, somatotopic correspondence with regions of the skin, while cells in the first stages of the auditory system are arranged tonotopically-that is, according to the auditory frequencies to which they react. Cells reacting to subtler properties of incoming stimuli are also arranged in regular geometries. For example, the angle of maximum response for orientation-sensitive cells of the visual cortex rotates systematically as one moves through small regions of the cortex; cells with corresponding retinal fields in the right and left eye reside in thin vertical strips of cortical tissue (ocular dominance columns) adjacent to, but sharply distinguished from, each other. Presumably these cell arrangements facilitate the interchange of information needed to detect significant features in incoming sensory streams, by highlighting sharp intensity and/or color changes, edge orientations, or sharp corners, for example. The picture suggested by the available evidence is one of the successive transformation of imagelike (because one- or two-dimensional) data structures to produce secondary imagelike structures in which stimulus features that are potentially useful for the formation of higher-level responses have been made explicit- a form of processing that is unsurprising in computer science terms. Even in the sensory realm, we do not know more than a few of the specific transformations that incoming data flows undergo, but we do know enough to think of this data and its processing in geometrically extended, imagelike terms. Beyond these early, relatively well-understood processing stages, one enters terra incognita, in which it has thus far proved impossible to correlate observed neural activity with any specific property of external stimuli.
Additional insight, consistent with the evidence just reviewed, comes from neuroembryology-from consideration of the pattern in which the cells of the brain knit themselves together. Neurons, like the cells constituting all other tissues, are initially motile that is, capable of migrating from their original positions, usually via a form of slow "walking" that is guided by the selective adhesiveness of a migrating cell to the tissues over which it pulls itself. This biochemically regulated cell motility plays a fundamental role in shaping the tissues and organs of the body during embryonic development: the sheets of cells that come to constitute these tissues are in many cases erected by the collective migration of their constituent cells, somewhat in the manner that a large circus tent can be erected by the collective motion of many people walking along under it and pulling on its expanding edges.
In neurons, however, similar patterns of motility act in a significantly different way. After the earliest phases of embryonic development, instead of the cell body itself migrating, a neuron throws out projections (its axons and dendrites to be), the ends of which carry small motile units known as growth cones. Each of these cones has twenty or so "feet" (pseudopodia) about one onethousandth of a millimeter in diameter and thirty times as long, which allow the cone to move over the surface of any tissue with which it comes in contact. The pseudopodia extend themselves in an apparently random manner from the growth cone in which they originate till they make contact with some nearby tissue surface. They then adhere to it with a force that is determined by the sugar- or starchlike side chains attached to modified protein molecules (glycoproteins), which are present in the cell membranes of the two contacting cells. Once contact is made, the pseudopodia contract, pulling the growth cone and the growing axon that develops behind this cone, apparently in the direction of the greatest adhesivity, just as a fly is forced to walk in the direction of maximal stickiness along flypaper on which it is trapped. Though other direction determining forces are undoubtedly involved, these adhesive effects, much elucidated through the brilliant work of Gerald Edelman and his collaborators at Rockefeller University, now appear basic not only to definition of interconnection patterns in the nervous system but also to embryological development in general.
Neurons' growth cones continue their walk, each apparently until it contacts any destination cell that is marked with some chemical substance to which the pseudopodia are sensitive, at which point some unknown enzymatic reaction destroys the capacity of the pseudopodia to continue moving. The growth cone then metamorphoses into a synapselike structure that subsequently develops into a mature synapse.
This picture of the nervous system's development does not suggest that the pattern of connections formed in complex mammalian brains can be entirely specific in the sense of creating perfectly determinate connections between specifically identified neurons, as if the neurons were transistors on an artificially engineered silicon chip, and the connections between them were formed by laying down metal in very precise fashion. Rather, this picture suggests a system, perhaps containing hundreds, thousands, even tens of thousands of neuron subspecies, possibly distinguishable biochemically and perhaps differing significantly in their detailed reaction to external stimuli but probably interconnected with relatively coarse specificity. The rules of growth that apply may, for example, only specify that a neuron of a particular type originating at a certain point in a particular brain layer will connect via an axon and a synapse to any neuron of some second type that is close to some other position in another brain region. Known growth mechanisms are sufficient to yield structures having this degree of specificity, but that they can produce the vastly more specific structures that characterize computer circuitry and that often enter into the neural system models of speculative thinkers coming from computer backgrounds seems doubtful.
It is worth noting, in this connection, that we presently lack not only detailed knowledge of the interconnection pattern of the brain but also comprehensive understanding of the more fundamental and rudimentary question of how many biochemically distinct species of neurons inhabit the brain. Partly for this reason, theorists who propose abstract brain models often begin by assuming that all neurons are functionally identical and accordingly model neurons as simple threshold elements that emit signals whenever the sum of their incoming excitatory stimuli, minus the sum of their incoming inhibitory stimuli, exceeds some fixed or adjustable threshold value. It is as if an investigator faced with the problem of analyzing an immense computing system of wholly unknown internal architecture began by assuming that all its integrated circuit chips were identical simply because at first glance they looked roughly the same and because any hypothesis closer to the truth was too dispiriting. Cursory microscopic examination of the brain's population of neurons, however, shows them to differ from one another as much as garden shrubs differ from giant redwoods. Moreover, even cells of apparently identical external morphology may differ biochemically in ways that cause their reactions to similar patterns of incoming stimuli to be widely different. Over the next decade or two, systematic use of the increasingly powerful battery of monoclonal antibodies available as ultrasensitive biochemical reagents will probably dispel much of our present ignorance about the varieties of neurons.
Nevertheless, while ignorance persists, the utility of even standard neuroanatomical information is compromised, since what is wanted is knowledge of the manner in which informationally significant (and, presumably, biochemically distinguishable) cell populations interconnect. In contrast, the information available concerns only the manner in which brain regions interconnect.
The massive cell death that occurs immediately after birth confirms the impression that the brain is designed to function correctly even if the neurons constituting it link up in a manner that is only approximately correct. It is well known that in newborn mammals some 15 percent of the neurons present in the neonate die out in early infancy. Evidence suggests that many of those neurons represent either overgrowth or neurons that for some reason have formed improper connections and are therefore failing to receive the electrical or chemical stimuli needed to keep them viable. Still further evidence suggesting that the interconnections in the brain are not entirely specific comes from experiments in which the synaptic connections made by a population of neurons are destroyed by cutting the axons connecting these synapses to their originating cell bodies. Such cutting usually causes the sprouting of nearby neurons and subsequently the formation of abnormal synaptic connections among neurons that would not ordinarily form connections in the affected brain region. This evidence suggests that a mechanism of competitive growth is involved in the development of interconnections among neurons and that neurons invade unoccupied synaptic space in much the same way that growing grasses tend to invade an initially empty field-not a situation that favors computerlike wiring precision.
These considerations suggest that the brain may be incapable of using the patterns of information processing that are most effective for artificial computers, even very large parallel computers. Artificial computing systems can often generate desired results most effectively (sometimes with remarkable efficiency) by using carefully designed and coordinated sequences of elementary processing steps. In such processing, the arrays of data being processed move through a sort of closely coordinated, massively parallel square dance, during which each data item interacts with all the items it encounters in such a way as to leave the desired output in place at the end of the procedure. Any failure in synchronization or in a local operation combining two operands when they meet generates a wave of error and leaves a meaningless result at the end. Such delicately balanced parallel processes only generate their intended results, or for that matter any useful result, if each motion of every one of the thousands of data items being processed takes place precisely at the moment specified for it and if every one of the millions of arithmetic or logical operations involved works perfectly. The evidence I have cited suggests that biological systems are not wired precisely enough to support this extremely delicate style of information processing. In particular, we have no evidence that the nervous system operates in other than perfect asynchrony, so that no form of information processing that requires close synchronization or that becomes substantially less expensive in its absence is an attractive candidate for use in neural systems.
Evolutionary considerations also suggest that the brain makes no use of delicately balanced processing patterns of the kind that are so common and effective in computer practice. Indeed, evolution proceeds by the accumulation of tiny random changes, each typically affecting but one detail of one of the thousands or tens of thousands of protein molecules whose interaction determines cell biology and function. For evolutionary pressures to carry change very far, each successive evolutionary step
must provide change-carrying organisms with enough of an advantage to favor their survival, at least marginally. This observation seems to rule out major qualitative jumps from an established pattern of activity to some other radically different and delicately balanced programlike pattern, no part of which is useful until an entire structure is put in place. Clever information-processing algorithms require exactly such complex interlocking logical constructions-another reason their use in a biological setting is inappropriate.
Our still insufficiently developed knowledge of neuroanatomy and the biochemistry of neurons does not provide the information that would enable us to model the activity of the nervous system at all specifically. Partly for this reason, theorists have remained attracted to homogeneous neural models and to highly conjectural (even if appealing) theories that the brain or important parts of it progress-from an informationally blank initial condition to a state in which much usable information is encoded-via a process of learning that acts at the synaptic level. The commonest theory of this type is one that Hebb initially proposed in the 1940s. According to Hebb, synapses receiving excitatory stimuli during periods in which the neuron to which they attach is active grow more sensitive and hence act more strongly on subsequent occasions to stimulate the firing of the same neuron. The efficacy of synapses not involved in a pattern of synaptic stimuli that repeatedly cause a cell to fire may then diminish in relative, perhaps also in absolute, terms; eventually, such
synapses become partly or wholly incapable of stimulating their cell. Hebb's proposed mechanism allows initially undifferentiated cells to become selectively conditioned to a variety of patterns that can originate directly in sensory systems or indirectly in the earlier stages of neural processing. His hypothesis has appealed to theoretically minded neural modelers, since it does not conflict with any available evidence yet suggests a way in which learning can mold neural structures about which little needs to be assumed. Moreover, it is easy to imagine biochemical mechanisms, compatible with Hebb's hypothesis, that might allow very powerful information processing capabilities, such as general forms of associative memory, to develop.
In spite of this appeal, only recently have we begun to have hard experimental evidence to support Hebb's conjecture, and this only for a very few regions of the brain, most notably the cerebellum. Recent research on this important brain structure shows that it functions, at least in small part, as a mechanism for storage of simple conditioned reflexes. This function has been demonstrated by showing that suitably patterned and intense simultaneous stimulation of appropriate neurons (specifically, the cerebellar "dimbing" fibers and the parallel fibers originating in the granule cells of the cerebellum) causes long-lasting changes in the sensitivity of the large cerebellar Purkinie cells. These are the same cells as are presumably involved in the formation of simple Pavlovian conditioned reflexes. It is possible Ln this way to establish a conditioned reflex (for example, conditionLng to a preceding auditory stimulus of the primitive eye-blink reflex Wriggered by a corneal air puff), even if one or both of the experimental factors (air puff or auditory stimulus) normally entering its formation is replaced by direct electrical stimulation of corresponding cerebellar input. The modifications that occur during the formation of such electrically induced artificial conditionings can be localized to a single class of synapse, namely the synapses between parallel fibers and the multiple Purkinje cell layers these fibers Taverse. This experimental work brilliantly verifies the theoretical conjectures concerning the role of the cerebellum that David Marr nd James Albus put forth years earlier. These conjectures were inspired by the striking abstract resemblance between cerebellar nicroanatomy and the physical layout of certain types of computer memory.
Beyond these profoundly intriguing, but still limited, insights, earning-based theories of the origin of neural function remain subject to the objection that we know hardly anything yet about the actual locus or mechanism of other memory storage within the brain and even less about the way memories are modified to accomplish abstract learning. Though it is widely believed that synapses represent he elementary loci of memory storage and that memory storage is somehow accomplished by modifying synaptic reactivity, we have not yet been able to develop much clear biochemical evidence to support this belief. Most of the evidence relating to mammalian brains is still very indistinct. For example, certain often-cited studies indicate simply that experimental rats raised in stimulating environments apparently develop larger numbers of synapses than rats imprisoned in stimulus-free environments. Moreover, the process of synaptic modification revealed in studies of simpler nervous systems (Eric Kandel's famous work on Aplysia, for example) is not specifically Hebbian. The governing synaptic changes seen in these investigations seem to occur in the transmitting (presynaptic) rather than in the receiving (postsynaptic) side of synapses and hence are not in agreement with the mechanism Hebb assumed. Thus, theorists who take some hypothesis about learning as their starting point are choosing to begin in a particularly dark area of neuroscience.
To function effectively, theorists with research backgrounds in computer science and artificial intelligence need to extract an appropriate notion of neurocompatibility from the diffuse mass of evidence coming from wet-lab neuroscience. This notion must both reflect any detailed knowledge of nervous system function that is likely to cast light on the information-processing activities of the nervous system and define constraints on the modes of neural processing that can best guide the theorist's attempts to guess what is going on. Thus far, the following dues seem most useful:
1. The nervous system must make use of highly parallel algorithms involving only very limited numbers of successive stages of transformation of incoming data streams.
2. The best-known stages of early sensory processing seem to involve successive transformations of imagelike data structures to highlight data features that are probably important to subsequent formation of higher-level responses. The layout of this data in the neural sheets that process it is often in correspondence with continuous, varying parameters inherent in the data being processed (retinal position or edge orientation in the case of the eye, for example, and pitch in the case of the auditory system).
3. Neurons differing in the information they extract from a common incoming data stream are observed in the sensory system. Their existence may point to the existence of morphologically similar, but biochemically distinct, neuron subspecies within local areas of tissue, which might serve to carry separate dimensions of an incoming information flow. The number of these informationally significant subspecies is unknown and may possibly be large. The manner in which such subspecies make connection with each other and the geometry of the dendrite and axon arborizations they form may be significant for the data transformations that are then realized.
4. The crude picture of neurons as devices that sum incoming excitatory and inhibitory signals and that pass along as much of this sum as exceeds an inherent threshold may need to be sophisticated to allow for complex time lags and nonlinear effects, easily allowed by the very complex internal biocyde of all cells, neurons included.
5. The level of wiring accuracy in the nervous system seems to be low, and it seems to make no use of processing steps that involve tight synchronization of data motion or highly artificial interconnection patterns. Forms of processing that would arise naturally in neuron populations, possibly consisting of multiple subspecies that are interconnected in ways determined by simple growth rules, are most appealing as conjectures.
NEURAL NET MACHINES
Besides reflecting a desire to give theoretical assistance to experimental neuroscientists in their search for the way the living brain functions, the rapidly growing involvement of computer scientists with neuroscience has a second motive. This is to use knowledge of the brain to guide the design of new, very highly parallel computers- the so-called neural net machines. Though by no means likely to yield results quickly or easily, the contributions of computer science to neuroscience will come to attain full scientific legitimacy. Whether today's neuroscience will guide computer design in the near term seems far more doubtful. A substantial list of arguments supports this judgment:
1. Even in regard to the best-understood sensory systems, little is yet known about the detailed workings of the brain. Of brain function outside the sensory systems, we know essentially nothing. Hence, any claim that a specific computer architecture imitates a neural system is pure conjecture.
2. The living nervous system and the patterned silicon networks constituting computers differ much as technologies. The nervous system is three-dimensional and unsynchronized; it must probably tolerate very high degrees of miswiring but can form tens or hundreds of thousands of connections to each of its active computational elements, the neurons. At least for the moment, electronic circuit patterns are largely restricted to the two-dimensional surfaces of silicon chips and also (except where special, very regular patterns are used) to a few tens of connections per active element and a few hundred per chip; these circuits can, however, be wired with nearly perfect precision so that they operate in close synchrony.
3. Computers can exploit any artificial pattern of hardware interconnection or software processing that the intellectual work of machine and algorithm designers brings to light. We have argued that only an unknown minuscule fraction of these processing patterns is available for the evolution-constrained activity of the living brain. It is revealing to note that all the current major projects to design and build large parallel machines make use of highly artificial structures for communication and processing. This remark applies equally well to the Thinking Machines Corporation's Connection Machine (hypercube and rectangular grid communication), to NASA's Massively Parallel Processor, to ICL's Digital Array Processor (rectangular grid), to Intel Corporation's Hypercube (hypercube communication), to IBM's RP3, and to New York University's Ultracomputer (omeganet communication). Moreover, computations on these machines regularly use highly artificial and efficient parallel algorithms, not procedures suggestive of the constraints likely to affect information processing in natural neural structures.
Thus, enthusiastic discussion envisaging vast potential for some obscurely characterized form of neural net machine (and especially proposals to build such devices) seems suspect. At any rate, no serious argument justifying such claims has as yet appeared. The difficulties encountered in past research certainly afford little encouragement.
However, one exception must be made to this reservation about X the prospect for neural analogies in electronic device design, in favor of the striking work on integrated electronic sensors of Carver Mead at the California Institute of Technology. Mead's idea can be put as follows. Although as used in digital circuitry, a single transistor accomplishes only the most elementary operations of Boolean logic (so that, for example, several dozen transistors are required to implement operations as simple as the addition of two decimal digits), much more is possible if the same transistor is used in analog rather than digital fashion. The analog use of circuitry treats circuit voltages as representations of numerical values, accurate to one or more decimal digits; digital use gives thresholds to these voltages by classifying them as high or low. The digital approach has become overwhelmingly popular because it decisively improves the logical stability of electronic computations and somewhat simplifies circuit fabrication, but the loss of information and of potential processing speed is substantial. If used in analog fashion, a few transistors can do arithmetic operations as complex as multiplication or extraction of logarithms, at least approximately; done digitally, the same operations require hundreds of transistors.
Though this potential advantage of analog computation has been well understood for decades, analog systems have steadily lost ground to their digital competitors. In the first place, the precision of digital systems can readily be extended to any desired level simply by adding as many digits as one likes to the representation of a numerical quantity. Only standard components of fixed cost are required. In contrast, the precision of analog systems is inherently limited by the accuracy with which their component devices can be fabricated and isolated from outside physical disturbances such as temperature changes. A consequence is that the cost of analog devices escalates very rapidly with each additional digit of precision required and soon reaches a limit of absolute infeasibility.
A second advantage of digital systems is that they can retain information with perfect accuracy for indefinitely long periods by storing it in devices of essentially perfect stability-in the now commonplace computer "memories." Since analog information (voltage values, for example) inevitably degrades and drifts with time, nothing directly corresponding is available in the analog sphere. Purely analog computers cannot, therefore, store their programs in the same sense that digital systems can. Hence, the control information for analog systems that are at all complex, plus any extensive tables of constants or auxiliary functions that may be necessary, must be stored digitally and converted to analog form when needed for analog computation. Still worse, all intermediate data must be reconverted to digital form if it is to be stored for any length of time. The clumsiness and inherent limitations of this situation have restricted analog computation to a steadily narrowing sphere until, at present, large-scale computation of this sort is almost nonexistent.
Mead's insight is that there is an important area in which the disadvantages of analog computation are irrelevant, namely, in the processing of streams of sensory information like audio information or moving images. Here the precision of digital systems is of little advantage, since conversion from some raw analog sensory form is required in any case, implying that the incoming data fed to an analysis system, whether digital or analog, will necessarily be of limited precision. Moreover, many of the common procedures in the initial processing of this data, and especially those standing in any conjectural relationship to initial sensory processing in the nervous system, make little or no use of prestored information, so the absence of memory in analog systems is not an objection. Consequently, there is reason to hope that analog networks can process sensory data in a manner that will profit from their great simplicity and compactness in relation to comparably functional digital systems. Mead has constructed two interesting systems that do so: a sound-spectrum analyzer modeled after the cochlear membrane of the inner ear, and an optical motion detector whose structure is similar to the retinal neuroanatomy of the eye. His work may suggest many other applications that allow combination of the performance advantage of analog computation with the extremely sophisticated, ultrahighdensity packaging that current very large-scale integration (VLSI) technology supports. It may inspire much imitation and open a new direction in electronic design.
Nevertheless, in my opinion, Mead's work is interesting as analog VLSI rather than silicon neuroscience; in particular, his soundspectrum analyzer models the mechanical structure of the inner ear rather than the neural structures that receive its outputs.
PERSPECTIVE
Stunning new discoveries can be expected from experimental neuroscience during the coming decade. The astonishing successes of molecular biology constitute one major ground for this optimistic assessment. Once the presence of some biochemically important, even if initially quite unknown, protein is suspected in a tissue of interest, molecular techniques can be used to produce substantial quantities of antibody to this protein. Once available ("raised," in the jargon of the molecular biologist), such an antibody (which is simply a protein containing a portion complementary to some molecular detail of the protein) detects the presence of its target protein with exquisite sensitivity. Moreover, the antibody can readily be marked radioactively, magnetically, or optically and can be used to make the particular cells containing the target protein, or even particular microanatomical features of these cells, visible under the electron microscope. In addition, the walls of glass tubes can be coated with the antibody, and these columns can then be used to concentrate the protein by factors of a million or more. Such concentration opens a path to the chemical and structural analysis of the protein and from this to the identification of biochemical antagonists to its normal activity. Then, by dosing living brain tissue with these antagonists, one can paralyze the portion of normal function that is mediated by the protein and thereby pinpoint its specific physiological role and relevance for the information-processing activity of the brain.
As investigations of this sort are pursued more and more comprehensively for the entire battery of proteins significant as surface receptors in neurons, cell populations will become identifiable by the collections and concentrations of receptor molecules that their surfaces carry. Moreover, we will come to know the manner and speed with which neurons respond to the activation of particular surface receptors. These responses may be fast and electrically mediated or slow and activated through long chains of intermediate biochemical effects triggered by initial receptor molecules.
As noted earlier, biochemical identification of the subpopulations of neurons resident in the brain will give new focus to the work of neuroanatomists, increasing the relevance of their painstaking tracing of the brain s internal connections to our understanding of the brain's information processing. Embryological studies of the specialization and migration of biochemically identified cell populations within the developing brain will reveal the mechanisms that guide the formation of these connection patterns and improve our sense of what information-processing schemes the brain's substructures use and how complex they are. New chemically and optically mediated techniques are already beginning to improve our ability to observe the brain's electrical activity directly. Now, for instance, we can record the electrical activity of up to several hundred interconnected cells simultaneously. At some point we will also gain insight into the specific biochemical mechanism (or perhaps the many mechanisms) underlying memory, and that will enable us to formulate far more specific models of the memory processes, be these Hebb-like or not.
Though these massive experimental efforts will involve thousands of biochemists and neuroscientists for many years, we can expect investigations like these as well as ingenious, entirely new techniques eventually to uncover a very revealing mass of detail concerning brain function. As this information surfaces, the present aspiration of computer scientists to integrate their knowledge with that of neuroscientists will grow in relevance. Those in the computer science community who have paid their dues to experimental neuroscience by digesting and tracking its mounting mass of information may then be able to play an important part in extracting broad systematizing principles from an initial forest of experimental detail. These will surely be insights standing at the very pinnacle of science.
Jacob T. Schwartz is a professor at the Courant Institute of Mathematical Sciences at New York University.
