Is forgetting from working memory (WM) better explained by decay or interference? The answer to this question is the topic of an ongoing debate. Recently, a number of studies showed that performance in tests of visual WM declines with an increasing unfilled retention interval. This finding was interpreted as revealing decay. Alternatively, it can be explained by interference theories as an effect of temporal distinctiveness. According to decay theories, forgetting depends on the absolute time elapsed since the event to be retrieved. In contrast, temporal distinctiveness theories predict that memory depends on relative time, that is, the time since the to-be-retrieved event relative to the time since other, potentially interfering events. In the present study, we contrasted the effects of absolute time and relative time on forgetting from visual WM, using a continuous color recall task. To this end, we varied the retention interval and the inter-trial interval. The error in reporting the target color was a function of the ratio of the retention interval to the inter-trial interval, as predicted by temporal distinctiveness theories. Mixture modeling revealed that lower temporal distinctiveness produced a lower probability of reporting the target, but no changes in its precision in memory. These data challenge the role of decay in accounting for performance in tests of visual WM, and show that the relative spacing of events in time determines the degree of interference.