We study some mechanisms responsible for synchronous oscillations and loss of synchrony at physiologically relevant frequencies (10-200 Hz) in a network of heterogeneous inhibitory neurons. We focus on the factors that determine the level of synchrony and frequency of the network response, as well as the effects of mild heterogeneity on network dynamics. With mild heterogeneity, synchrony is never perfect and is relatively fragile. In addition, the effects of inhibition are more complex in mildly heterogeneous networks than in homogeneous ones. In the former, synchrony is broken in two distinct ways, depending on the ratio of the synaptic decay time to the period of repetitive action potentials (tau s/T), where T can be determined either from the network or from a single, self-inhibiting neuron. With tau s/T > 2, corresponding to large applied current, small synaptic strength or large synaptic decay time, the effects of inhibition are largely tonic and heterogeneous neurons spike relatively independently. With tau s/T < 1, synchrony breaks when faster cells begin to suppress their less excitable neighbors; cells that fire remain nearly synchronous. We show numerically that the behavior of mildly heterogeneous networks can be related to the behavior of single, self-inhibiting cells, which can be studied analytically.