Information from long-term memory is used to identify appropriate responses to cues in the environment. Left ventrolateral prefrontal cortex (VLPFC) has been implicated in the effortful retrieval of semantic representations, as well as in the goal-directed selection between such representations. It has also been suggested that left posterior middle temporal gyrus (pMTG) stores the rules which VLPFC accesses to guide behavior. In the present event-related fMRI study, we examined the contributions of left VLPFC and pMTG in the controlled retrieval and selection of action-relevant knowledge associated with road signs. Controlled retrieval demands were manipulated by varying how recently the sign meaning was learned, and selection demands were manipulated by varying the number of competing meanings associated with a sign. Activation in anterior VLPFC was consistent with controlled retrieval, activation in posterior VLPFC was consistent with selection, and activation in mid-VLPFC was sensitive to both manipulations. Left pMTG, while active, was not sensitive to these manipulations. These findings highlight the role of left VLPFC in accessing and maintaining goal-relevant information for the control of action.