Apolipoprotein E ε4 (ApoE4) has been implicated as a potential genetic risk factor for dementia. In this study, we investigate the effect of ApoE4 on learning and memory, changes in brain volume and neuroinflammatory responses in brain of ApoE4 transgenic mice. Four groups of male mice with ApoE4 and age-matched wild type (WT) (6-, 12-, 18- and 24-month) were studied. Spatial learning and retaining of mice was examined in the Morris Water Maze (MWM). Changes in brain volume (including the whole brain, hippocampus, cortex, total ventricles, and caudate putamen) were assessed by using 7T small animal MRI. Neuroinflammatory responses were analyzed by measuring the levels of microglia (Iba-1), iNOS, TNFα, and IL-6 quantitatively. In the MWM, ApoE4 mice showed longer escape latency (p < 0.05) and swim distance (p < 0.05) at age 12 month and older, comparing with the WT mice. They also demonstrated poor memory retention in the probe test (p < 0.05). Brain atrophy was significant in ApoE4 mice than age-matched WT mice (18 months: 0.079 ± 0.004 versus 0.086 ± 0.003, p = 0.018; and 24 months: 0.074 ± 0.005 versus 0.084 ± 0.006, p = 0.008). The expression of Iba-1, iNOS, and TNFα in hippocampus and cortex were significantly higher in ApoE4 mice than in WT mice at 12 months and older. These data suggest that ApoE4 plays an important role in learning and memory impairment. These deficits are associated with neuroinflammatory responses that may in turn lead to atrophy in hippocampus and cortex.