We asked human observers to adjust the color of natural fruit objects until they appeared achromatic. The objects were generally perceived to be gray when their color was shifted away from the observers' gray point in a direction opposite to the typical color of the fruit. These results show that color sensations are not determined by the incoming sensory data alone, but are significantly modulated by high-level visual memory.