1. The mechanisms by which 5-hydroxytryptamine (5-HT) depresses transmitter release from lamprey reticulospinal axons were investigated. These axons make glutamatergic synapses onto spinal ventral horn neurons. 5-HT reduces release at these synapses, yet the mechanisms remain unclear. 2. Excitatory postsynaptic currents (EPSCs) evoked by stimulation of reticulospinal axons were recorded in ventral horn neurons. 5-HT depressed the EPSCs in a dose-dependent manner with an apparent Km of 2.3 microM. 3. To examine the presynaptic effect of 5-HT, electrophysiological and optical recordings were made from presynaptic axons. Action potentials evoked Ca(2+) transients in the axons loaded with a Ca(2+)-sensitive dye. 5-HT slightly reduced the Ca(2+) transient. 4. A third-power relationship between Ca(2+) entry and transmitter release was determined. However, presynaptic Ca(2+) currents were unaffected by 5-HT. 5. Further, in the presence of a K(+) channel blocker, 4-aminopyridine (4-AP), 5-HT left unaltered the presynaptic Ca(2+) transient, ruling out the possibility of its direct action on presynaptic Ca(2+) current. 5-HT activated a 4-AP-sensitive current with a reversal potential of -95 mV in these axons. 6. The basal Ca(2+) concentration did not affect 5-HT-mediated inhibition of release. Although 5-HT caused a subtle reduction in resting axonal [Ca(2+)]i, synaptic responses recorded during enhanced resting [Ca(2+)]i, by giving stimulus trains, were equally depressed by 5-HT. 7. 5-HT reduced the frequency of TTX-insensitive spontaneous EPSCs at these synapses, but had no effect on their amplitude. We propose a mechanism of inhibition for transmitter release by 5-HT that is independent of presynaptic Ca(2+) entry.