The main finding of this paper is that the human visual cortex responds in a very nonlinear manner to the color contrast of pure color patterns. We examined human cortical responses to color checkerboard patterns at many color contrasts, measuring the chromatic visual evoked potential (cVEP) with a dense electrode array. Cortical topography of the cVEPs showed that they were localized near the posterior electrode at position Oz, indicating that the primary cortex (V1) was the major source of responses. The choice of fine spatial patterns as stimuli caused the cVEP response to be driven by double-opponent neurons in V1. The cVEP waveform revealed nonlinear color signal processing in the V1 cortex. The cVEP time-to-peak decreased and the waveform's shape was markedly narrower with increasing cone contrast. Comparison of the linear dynamics of retinal and lateral geniculate nucleus responses with the nonlinear dynamics of the cortical cVEP indicated that the nonlinear dynamics originated in the V1 cortex. The nature of the nonlinearity is a kind of automatic gain control that adjusts cortical dynamics to be faster when color contrast is greater.