Insomnia is the most frequently encountered sleep complaint worldwide. While many prescription drugs are used to treat insomnia, extracts of valerian (Valeriana officinalis L., Valerianaceae) are also used for the treatment of insomnia and restlessness. To determine novel mechanisms of action, radioligand binding studies were performed with valerian extracts (100% methanol, 50% methanol, dichloromethane [DCM], and petroleum ether [PE]) at the melatonin, glutamate, and GABA(A) receptors, and 8 serotonin receptor subtypes. Both DCM and PE extracts had strong binding affinity to the 5-HT(5a) receptor, but only weak binding affinity to the 5-HT(2b) and the serotonin transporter. Subsequent binding studies focused on the 5-HT(5a) receptor due to the distribution of this receptor in the suprachiasmatic nucleus of the brain, which is implicated in the sleep-wake cycle. The PE extract inhibited [(3)H]lysergic acid diethylamide (LSD) binding to the human 5-HT(5a) receptor (86% at 50 microg/ml) and the DCM extract inhibited LSD binding by 51%. Generation of an IC(50) curve for the PE extract produced a biphasic curve, thus GTP shift experiments were also performed. In the absence of GTP, the competition curve was biphasic (two affinity sites) with an IC(50) of 15.7 ng/ml for the high-affinity state and 27.7 microg/ml for the low-affinity state. The addition of GTP (100 microM) resulted in a right-hand shift of the binding curve with an IC(50) of 11.4 microg/ml. Valerenic acid, the active constituent of both extracts, had an IC(50) of 17.2 microM. These results indicate that valerian and valerenic acid are new partial agonists of the 5-HT(5a) receptor.