Although the majority of ovarian follicles undergo atresia through a mechanism involving apoptotic cell death, in vivo studies concerning the hormonal regulation of atresia have been difficult due to the presence of heterogeneous population of follicles in the ovary. In the present study, the regulation of follicle apoptosis by gonadotropins, insulin-like growth factor I (IGF-I), and IGF-binding protein 3 (IGFBP-3) was examined using a serum-free culture of preovulatory follicles. Immature rats at 26 days of age received a single dose of PMSG. Two days later, the largest preovulatory follicles were collected for in vitro culture with or without hormones. After 24 h of culture, follicular apoptotic DNA fragmentation was analyzed by autoradiography of size-fractionated DNA labeled at 3'-ends by [32P]dideoxy-ATP. A spontaneous increase in apoptotic DNA fragmentation occurred after 24 h of culture in the absence of hormones, whereas treatment with human CG (hCG) or FSH suppressed follicular apoptosis in a dose-dependent manner, with 0.1 microgram/ml causing maximal suppression by 60-62%. Cotreatment with hCG and FSH had no additional effect. Like gonadotropins, treatment with IGF-I and insulin also suppressed the spontaneous onset of apoptosis, with IGF-I being more effective than insulin. Cotreatment with IGFBP-3 and hCG dose-dependently reversed the suppressive effect of hCG on apoptosis by 42%, suggesting a mediatory role of endogenously produced IGF-I. The addition of IGFBP-3 also blocked the suppressive action of IGF-I by 49%, whereas it did not affect the suppressive action of an IGF-I agonist or insulin. Treatment with IGFBP-3 alone had no effect on apoptotic DNA fragmentation. Estrogen and progesterone production by the cultured follicles were also analyzed by RIA. Gonadotropin treatment resulted in a marked stimulation of the production of both steroid productions. In contrast, treatment with IGF-I caused a small increase in estrogen but decreased progesterone production. Although treatment with IGFBP-3 alone decreased both estrogen and progesterone production, cotreatment with IGFBP-3 and hCG resulted in a slight decrease in estrogen production but an increase in progesterone production. Furthermore, IGFBP-3 did not affect IGF-I action on steroid production. To further substantiate the hypothesis that IGFBP-3 blocks the suppressive effect of hCG on apoptosis by neutralizing endogenously produced IGF-I, solution hybridization analysis was performed, and hCG treatment was shown to increase IGF-I messenger RNA levels in cultured follicles by 1.9-fold.(ABSTRACT TRUNCATED AT 400 WORDS)