To make deliberate decisions, we have to utilize detailed information about the environment and our internal states. The ventral visual pathway provides detailed information on object identity, including color and shape, to the ventrolateral prefrontal cortex (VLPFC). The VLPFC also receives motivational and emotional information from the orbitofrontal cortex and subcortical areas, and computes the behavioral significance of external events; this information can be used for elaborate decision making or design of goal-directed behavior. In this review, we discuss recent advances that are revealing the neural mechanisms that underlie the coding of behavioral significance in the VLPFC, and the functional roles of these mechanisms in decision making and action programming in the brain.