We seek a general approach to determine what stimulus features visual neurons are sensitive to and how those features are represented by the neuron's responses. Because lesions of inferior temporal (IT) cortex interfere with a monkey's ability to perform pattern discrimination tasks we studied IT neurons. Previous single-unit studies have shown that IT neurons sometimes respond more strongly to complex stimuli (brushes, hands, faces) than to simple stimuli (bars, slits, edges). However, it is not known how specific stimulus parameters are represented by responses. We studied the responses of IT neurons in alert behaving monkeys to a large set of two-dimensional black and white patterns. The stimulus set was based on 64 Walsh functions that can be used to represent any picture with a resolution of one part in eight along each of two dimensions. The responses to these stimuli spanned a continuum from inhibition to strong excitation. A statistical test showed that the spike count was determined by which Walsh stimulus was presented. Hence, these stimuli form an adequate set for testing IT neurons. The responses showed temporal modulation of the spike train that could not be represented by a change in the spike count alone. Examples of this modulation were changes in latency, changes in the duration of the response, and alternating periods of excitation and inhibition. This temporal modulation may be important in representing stimulus parameters. The next paper in this series develops a method for quantifying this temporal modulation and shows that it is dependent on the stimulus. The third paper in this series shows that this temporal modulation contains more information about stimulus parameters than is contained in the spike count alone.