A clonal cell line derived from rat renal mesangial cells was shown to express endogenous 5-hydroxytryptamine (serotonin, 5-HT) receptors that mediate inhibition of cyclic AMP accumulation. These receptors were characterized as being of the 5-HT1B receptor subtype. 5-HT1 receptor agonists inhibited forskolin-stimulated cyclic AMP accumulation in rat renal mesangial cells (60-70% maximal inhibition) with the following rank order of potency (mean pEC50 values +/- SEM, n > or = 3): ergotamine (9.58 +/- 0.51) > RU 24969 (8.67 +/- 0.23) > or = 5-CT (8.42 +/- 0.06) > or = CP 93129 (8.15 +/- 0.27) > 5-HT (7.75 +/- 0.11) > sumatriptan (6.29 +/- 0.30) > 8-OH-DPAT (4.32 +/- 0.15). 5-HT2 and 5-HT4 receptor agonists were without effect. 5-HT-induced inhibition of cyclic AMP accumulation was abolished by a pre-treatment of the cells with pertussis toxin. (-)Propranolol was a partial agonist (27% maximal inhibition, pEC50 7.19 +/- 0.24, n = 3); when used as an antagonist at 1 microM, it shifted the concentration-response curve of 5-HT to the right (pKB 7.22 +/- 0.35, n = 3). Methiothepin was a competitive antagonist of 5-HT (pA2 8.04 +/- 0.10, Schild slope 0.87 +/- 0.21, n = 3). Rauwolscine (10 microM) had no antagonist activity. There was a significant correlation (r = 0.98, P = 0.0001) between the cyclic AMP data obtained in rat mesangial cells and 5-HT1B binding data reported in rat brain cortex. The same pattern of responses was observed in early passages of primary cultures of rat mesangial cells.(ABSTRACT TRUNCATED AT 250 WORDS)