Both 5-HT3 receptor antagonists and benzodiazepine receptor ligands have effects on anxiety, and alter the behavioral action of ethanol. For these reasons, we tested the ability of several 5-HT3 receptor antagonists to inhibit the ligand binding and function of the gamma-aminobutyric acidA/benzodiazepine receptor Cl- channel complex of mouse brain membranes. MDL 72222 (1-a-H-3-a-5-aH-optropan-3yl-3,5-dichlorobenzoate) and LY 278584 (1-methyl-N-(8-methyl-8-azabicyclo[3.2.1.]oct-3-yl)-1H-indazole-3- carboxamide) inhibited [3H]flunitrazepam binding with Ki values of approximately 20 microM; ICS 205-930 (3 alpha-tropanyl-1H-indole-3-carboxylic acid ester) was more potent with a Ki of 0.8 microM. ICS 205-930 (50 microM) had no effect on [3H]muscimol binding. ICS 205-930, MDL 72222, and LY 278584 all inhibited the binding of [35S]TBPS (tert-butylbicyclophosphorothionate) with Ki values of approximately 10 microM and reduced muscimol-dependent 36Cl- flux into mouse cortical microsacs by 30-45% at a concentration of 10 microM. ICS 205-930, MDL 72222, and LY 278584 (at micromolar concentrations) reduced GABA-gated chloride currents studied in Xenopus oocytes expressing human alpha 1 beta 1 gamma 2S GABAA receptor subunits. ICS 205-930 differed from the other two 5-HT3 receptor antagonists in that it induced a biphasic effect on GABA-gated currents: at concentrations from 0.1 to 5 microM it potentiated GABA responses, whereas at higher concentrations (50-100 microM) it produced inhibition. The stimulatory action induced by ICS 205-930 was due to interaction at the benzodiazepine recognition site because expression of the gamma 2 subunit was required and Ro 15-1788 (1 microM) completely prevented the potentiation caused by ICS 205-930. Thus, several 5-HT3 receptor antagonists inhibit benzodiazepine binding and affect GABAA receptor function. These actions are most pronounced for ICS 205-930 and likely involve direct affects on the GABA/benzodiazepine complex rather than interactions with 5-HT3 receptors.