Dissociable Effects of Selective 5-HT_2A and 5-HT_2C Receptor Antagonists on Serial Spatial Reversal Learning in Rats

Abstract

Serotonin (5-hydroxytryptamine, or 5-HT) is strongly implicated in the ability to shift behavior in response to changing stimulus-reward contingencies. However, there is little information on the contribution of different 5-HT receptors in reversal learning. Thus, we investigated the effects of systemic administration of the 5-HT_2A antagonist M100907 (0, 0.01, 0.03, and 0.1 mg/kg, i.p.) and the 5-HT_2C antagonist SB 242084 (0, 0.1, 0.3, and 1.0 mg/kg, i.p.) on the performance of an instrumental two-lever spatial discrimination and serial spatial reversal learning task, where both levers were presented and only one was reinforced. The rat was required to respond on the reinforced lever under a fixed ratio 3 schedule of reinforcement. Following attainment of criterion, a series of within-session reversals was presented. Neither M100907 nor SB 242084 altered performance during spatial discrimination and retention of the previously reinforced contingencies. M100907 significantly impaired reversal learning by increasing both trials to criterion (only at the highest dose) and incorrect responses to criterion in Reversal 1, a pattern of behavior manifested as increased perseverative responding on the previously reinforced lever. In contrast, SB 242084 improved reversal learning by decreasing trials and incorrect responses to criterion in Reversal 1, with significantly fewer perseverative responses. These data support the view that 5-HT_2A and 5-HT_2C receptors have distinct roles in cognitive flexibility and response inhibition. The improved performance in reversal learning observed following 5-HT_2C receptor antagonism suggests these receptors may offer the potential for therapeutic advances in a number of neuropsychiatric disorders where cognitive deficits are a feature, including obsessive-compulsive disorder.

INTRODUCTION

Cognitive inflexibility, the inability spontaneously to withhold, modify, or sustain adaptive behavior in response to changing situational demands, is associated with various psychiatric disorders, most notably schizophrenia, depression, and obsessive-compulsive disorder (OCD). The elucidation of the underlying neurochemical mechanisms of cognitive flexibility and constituent processes including response inhibition could be of major importance for the understanding of the etiology and treatment of inflexible behavior apparent in such disorders.

The reversal learning task has been used as a measure of behavioral flexibility in humans (Rolls et al, 1994; Rogers et al, 2000; Murphy et al, 2002; Fellows and Farah, 2003), nonhuman primates (Jones and Mishkin, 1972; Butter, 1969; Dias et al, 1996; Clarke et al, 2004, 2005, 2007; Lee et al, 2007), and rats (Birrell and Brown, 2000; McAlonan and Brown, 2003; Idris et al, 2005; van der Meulen et al, 2006; Boulougouris et al, 2007). Converging evidence from a number of studies has implicated the orbitofrontal cortex (OFC) (human, Rolls et al, 1994; nonhuman primate, Settlage et al, 1948; Jones and Mishkin, 1972; Butter, 1969; Dias et al, 1996; rat, Chudasama and Robbins, 2003; McAlonan and Brown, 2003; Boulougouris et al, 2007) and the ventrolateral sector of caudate nucleus (monkey, Divac et al, 1967; rat, Dunnett and Iversen, 1980), while a recent study demonstrated that basolateral amygdala lesions abolished OFC-induced reversal learning impairments (Stalnaker et al, 2007). Efficient reversal learning calls upon specific operations such as (1) detection of the shift in contingency; (2) inhibition of a prepotent, learned response; (3) overcoming ‘learned irrelevance’; and (4) new associative learning.

Serotonin (5-hydroxytryptamine, or 5-HT) is a monoamine neurotransmitter that has been strongly implicated in behavioral flexibility. Selective 5-HT depletions in the marmoset prefrontal cortex induced by the neurotoxin 5,7-dihydroxytryptamine (5,7-DHT) impaired performance on a serial visual discrimination reversal learning task, which was mainly due to perseverative responding to the previously rewarded stimulus (Clarke et al, 2004). Subsequent work has established that this deficit was specific to reversal learning and not attentional set shifting (Clarke et al, 2005). More recently, it has been demonstrated that this deficit in reversal learning was specific to 5-HT and not dopamine (DA) depletion in the OFC (Clarke et al, 2007). Similarly, systemic administration of the 5-HT_1A receptor agonist 8-OH-DPAT impaired serial reversal learning by enhancing perseverative tendencies, an effect which was reversed by the selective 5-HT_1A receptor antagonist WAY100635 (Clarke et al, 2003). Deficits were found in 5-HT-deficient rats following a tryptophan-deficient diet and monkeys with high doses of the 5-HT₃ antagonist ondansetron (Barnes et al, 1990; Domeney et al, 1991). However, low doses of ondansetron (Domeney et al, 1991) and lysergic acid diethylamide (LSD) improved reversal learning (King et al, 1974), although these effects were not specific to reversal learning per se.

While the involvement of 5-HT systems in reversal learning is thus established, the particular 5-HT receptor subtypes that underlie these effects are not well understood. A growing body of evidence suggests that 5-HT_2A and 5-HT_2C receptors have opposing functional roles. For example, 5-HT_2C receptors appear to inhibit DA release, whereas activation of 5-HT_2A receptors enhances it (Millan et al, 1998; Di Matteo et al, 2001, 2002). Moreover, antagonism of 5-HT_2C receptors potentiates some of the behavioral effects of cocaine, whereas antagonism of 5-HT_2A receptors attenuates both cocaine-induced hypermotility and reinstatement of cocaine-seeking (Cunningham et al, 1992; Fletcher et al, 2002). Decreasing 5-HT transmission through blockade of 5-HT_2C receptors could therefore have opposing effects on behavior to those obtained through antagonizing 5-HT_2A receptors. With respect to inhibitory response control, recent reports indicate that the 5-HT_2C receptor antagonist SB 242084 increases premature responding on the five-choice serial reaction time task (5CSRTT), whereas 5-HT_2A receptor antagonist M100907 decreases the same measure (Higgins et al, 2003; Winstanley et al, 2004). In conclusion, although 5-HT_2A and 5-HT_2C receptors share similar pharmacological profiles with the highest degree of sequence homology (about 50% overall sequence identity), the apparently different behavioral actions elicited by antagonism at these receptors may be attributable to fundamental differences in signal transduction pathways of the two receptor subtypes (Berg et al, 1994, 1998).

The aim of the present study was to investigate the contribution of 5-HT_2A and 5-HT_2C receptors on the performance of rats in the instrumental two-lever spatial discrimination and serial reversal learning task through systemic administration of the selective 5-HT_2A receptor antagonist M100907 (Kehne et al, 1996) and the selective 5-HT_2C receptor antagonist SB 242084 (Kennett et al, 1997) to provide a direct comparison of these two 5-HT receptor subtypes on ‘cognitive flexibility’ and constituent processes including response inhibition.

MATERIALS AND METHODS

Subjects

Sixty-eight experimentally naive adult male Lister Hooded rats (Charles River, UK) weighting 280–320 g at the start of experiments were pair-housed under a reversed light cycle (lights on from 1900 to 0700 hours). Prior to the beginning of training, rats were handled for ≈5 min daily for 5 days and were put on to a food-restriction schedule (18 g of Purina lab chow per day). Water was available ad libitum and testing took place between 1300 and 1600 hours 7 days per week. One animal was excluded due to computer failure during testing. The work was carried out under a UK Home Office Project license (PPL 80/1767) in accordance with the UK Animals (Scientific Procedures) Act 1986.

Behavioral Apparatus

The behavioral apparatus consisted of eight operant conditioning chambers (30 × 24 × 30 cm; Med Associates, Georgia, VT), each enclosed within a sound-attenuating wooden box fitted with a fan for ventilation and masking of extraneous noise. Each chamber was fitted with two retractable levers located on either side of a centrally positioned food magazine, into which an external pellet dispenser could deliver 45 mg sucrose pellets (Noyes dustless pellets; Sandown Scientific, Middlesex, UK), a light-emitting diode (LED), which was positioned centrally above each lever, a magazine light, and a houselight. Magazine entry was detected by an infrared photocell beam located horizontally across the entrance. The apparatus was controlled by Whisker control software (www.whiskercontrol.com) and the task was programmed in Visual C++ (v.6).

Behavioral Procedure

Rats were trained on the instrumental two-lever spatial discrimination and serial reversal learning task as described and illustrated previously (Boulougouris et al, 2007). Briefly, rats were initially trained to nose poke in the central magazine to trigger presentation of the retractable levers and to respond on them under a fixed ratio 3 (FR-3) schedule for food delivery (pretraining). The FR-3 schedule was used to preclude the possibility of reinforcing single, accidental presses on the correct lever.

Acquisition of spatial discrimination

Training continued with the acquisition of a two-lever discrimination task. Now both levers were presented at trial onset and the rat had to learn that three lever presses on only one of these levers would result in reward.

Each session lasted 20 min and consisted of a maximum of five 10-trial blocks. Each trial began with the presentation of both levers and a visual stimulus (a lit LED). The lit LED was used as a distractor and its location (left/right) varied from trial to trial according to a pseudorandom schedule so that the light was presented an equal number of times on each side for the session. Thus, the only stimulus with informational value for the discrimination at this phase was the spatial position of the retractable levers. Throughout the session, three lever presses on one lever (lever A) would produce a single pellet reward and the retraction of both levers, whereas three responses on lever B would result in lever retraction without reward delivery. The position of the reinforced lever (left or right) was kept constant for each rat but was counterbalanced between subjects.

Each rat had one training session per day and was trained to a criterion of nine correct responses in one block of 10 trials (binomial distribution p<0.01, likelihood of attaining criterion in a 10-trial block). Once this criterion was reached, this initial discrimination phase was considered complete, and the animal was returned to the home cage. If the criterion was not achieved this phase was repeated the next day till criterion attainment (Figure 1).

Figure 1

Flow diagram of the behavioral procedure. Rats responded to levers under a fixed ratio (FR-3) schedule to obtain a pellet reward. The (√) and (×) symbols indicate which lever was correct and incorrect at each stage. The correct lever was counterbalanced across rats.

Within-session serial reversal learning task

In the next training session, reversal learning was introduced. By definition, reversal learning presupposes retention of a previously acquired discrimination. In serial reversals, in the first instance this would involve recall of the initially acquired discrimination described above. In subsequent reversals it would involve retention of the preceding reversal phase.

Accordingly, in the reversal session, animals were again exposed to the initial discrimination task described above (with the same lever rewarded as before: discrimination retention in the first instance, latest reversal retention in subsequent runs). This initial retention phase preceding reversal also comprised a maximum of five 10-trial blocks and once the criterion of nine correct responses in a 10-trial block was achieved, the position of the reinforced lever was reversed (reversal phase). The reversal phase also consisted of a maximum of five 10-trial blocks. The learning criterion was the same as in the initial phase (nine correct responses in a 10-trial block). Animals required more than one session to reach criterion on reversal phase. Thus, they received multiple, separate training sessions that were summed together to produce the final results. During these sessions the initial contingency was determined by retention fluency. For example:

All rats always achieved criterion in the initial retention phase preceding reversal.

A series of three reversals was given. Between successive reversals, animals were always given a single intervening day session of up to five 10-trial blocks where they were required to show retention of the previous reversal phase by reaching the 9 of 10 correct criterion in one 10-trial block (retention phase without reversal: same procedure as acquisition of spatial discrimination described above; Figure 1).

Drugs

M100907 and SB 242084 (Solvay, Weesp, The Netherlands) were tested in two different experiments. Prior to drug administration, animals were divided into four groups for each experiment, matched for their performance during the acquisition of the spatial discrimination. Each group received i.p. injections of either M100907 (0, 0.01, 0.03, and 0.1 mg/kg; Experiment 1) or SB 242084 (0, 0.1, 0.3, and 1.0 mg/kg; Experiment 2). All drugs were administered daily 20 min prior to the start of the behavioral task. Following initiation of drug testing, animals required 8–11 days to achieve criterion in Reversals 1–3.

M100907 was dissolved in saline and the pH adjusted to 6.25 using 0.1 M NaOH and 0.1 M HCl. SB 242084 was dissolved in 25 mM citric acid in 8% cyclodextrine in 0.9% saline, and the pH adjusted to 6.4 using 0.1 M NaOH. Systemic injections of drug were given in a volume of 1 ml/kg. Determination of doses was based on previous studies using the same drugs (Jones et al, 2002; Winstanley et al, 2004).

Statistical Analysis

The main measures of the animals' ability to learn the discrimination and reversals were: (1) the number of trials to criterion, (2) the total number of incorrect responses to criterion on completed (correct and incorrect) trials, and (3) the total number of errors (ie incorrect trials) to criterion. Type of errors were further analyzed as described previously (Boulougouris et al, 2007) according to the method of Dias et al (1996) and Bussey et al (1997), modified from Jones and Mishkin (1972). In this analysis, errors during reversal learning were broken down into two learning stages: errors committed before the attainment of chance level performance (<50% correct trials) and errors committed above chance (⩾50% correct trials). Jones and Mishkin regarded errors made during the first stage of learning as indicative of perseverative responses to the previously reinforced stimulus. Thus, stage 1 errors are termed ‘perseverative errors’ whereas stage 2 errors are termed ‘learning errors’. Additional secondary measures recorded for each trial were (3) the latency to respond, (4) the latency to collect the reward, (5) the number of omissions, and (6) the number of extra responses (ie incorrect responses in a trial scored as correct).

Data for each variable were subjected to a repeated-measures ANOVA. Where significant interactions were detected, they were further explored through separate ANOVAs or planned comparisons (contrast testing) to establish simple effects. For all comparisons, significant difference was assumed at p<0.05. The between-subject factor was Drug (four levels: three different doses of drug plus vehicle) and the within-subject factors were either Retention phase without reversal occurring (three levels: retention of spatial discrimination, retention of Reversals 1–2), or Retention phase preceding reversal (three levels: retention of spatial discrimination, retention of Reversals 1–2), or Reversal phase (three levels: Reversals 1–3).

RESULTS

Prior to drug administration, the groups did not differ in the number of incorrect responses to reach the performance criterion on the acquisition of spatial discrimination (M100907: F_3,30=1.10, p=0.37; SB 242084: F_3,29=0.295, p=0.83; data not shown).

M100907 had no significant effects, at any dose, on retention (with or without reversal occurring) of the drug-free spatial discrimination or the previously acquired reversals, as indicated by a lack of effect on the number of trials to reach criterion (Figure 2a) or the number of incorrect responses (Figure 3a). However, rats treated with M100907 at the two highest doses (0.03 and 0.1 mg/kg) exhibited significant impairment of performance in reversal learning. Specifically, M100907 (0.1 mg/kg) significantly increased trials to criterion in reversal phase 1 (reversal 1—vehicle vs 0.1 mg/kg contrast: F_1,30=15.80, p=0.0004; Figure 2b), whereas both doses of 0.03 and 0.1 mg/kg significantly increased incorrect responses to criterion (reversal 1—vehicle vs 0.03 mg/kg contrast: F_1,30=4.93, p=0.03; vehicle vs 0.1 mg/kg contrast: F_1,30=19.81, p<0.001; Figure 3b). Animals treated with the two highest doses of M100907 made significantly more perseverative errors (ie <50% correct) than controls in reversal 1 phase (reversal 1—vehicle vs 0.03 mg/kg: F_1,30=6.75, p=0.014; vehicle vs 0.1 mg/kg: F_1,30=18.75, p<0.001; Figure 4). No differences were noted in learning errors (Figure 4).

Figure 2

Number of trials to criterion through (a) the retention phase (without reversal) and retention (initial) phase (preceding reversal) and (b) Reversal phase of Experiment 1. Data are presented as mean values±SEM (a) Retention phase (without reversal): there was no significant main effect of group (F_3,30=0.6, p=0.62), a significant main effect of retention phase (F_2,60=16.92, ^***p<0.001), and no significant group × retention phase interaction (F_6,60=1.37, p=0.24). Retention phase (preceding reversal): there was no significant main effect of group (F_3,30=0.08, p=0.97), a significant main effect of retention phase (F_2,60=51.83, ^***p<0.001), and no significant group × retention phase interaction (F_6,60=0.79, p=0.59). (b) Reversal phase: there were significant main effects of group and reversal phase (F_3,30=5.63, ^**p=0.0035 and F_2,60=129.7, ^***p<0.001, respectively) and a significant group × reversal phase interaction (F_6,60=2.91, ^*p=0.0148). Planned comparisons demonstrated that only the highest dose of M100907 (0.1 mg/kg) significantly increased trials to criterion compared with the vehicle group in reversal phase 1 (F_1,30=15.80, p=0.0004). Asterisks above bars denote significant differences (ANOVA; ^***p<0.001) from vehicle-treated controls.

Figure 3

Incorrect responses of completed trials through (a) the retention phase (without reversal) and retention (initial) phase (preceding reversal) and (b) Reversal phase of Experiment 1. (a) Retention phase (without reversal): there were no significant differences between the groups in their ability to retain the drug-free acquired spatial discrimination or to retain previously acquired reversals (group: F_3,30=0.71, p=0.56; retention phase: F_2,60=22.25, ^***p<0.001; group × retention phase: F_6,60=0.93, p=0.48). Retention phase (preceding reversal): there was no significant main effect of group (F_3,30=0.82, p=0.49), a significant main effect of retention phase (F_2,60=23.21, ^***p<0.001), and no significant group × retention phase interaction (F_6,60=1.34, p=0.25). (b) Reversal phase: analysis yielded significant main effects of group and reversal phase (F_3,30=6.0, ^**p<0.003 and F_2,60=226.6, ^***p<0.001, respectively) and a significant reversal phase × group interaction (F_6,60=4.4, ^***p<0.001). Contrast testing revealed that rats treated with the two highest doses of M100907 (0.03 and 0.1 mg/kg) made significantly more incorrect responses than controls in reversal 1 phase (F_1,30=4.93, ^*p=0.03 and F_1,30=19.81, ^***p<0.001, respectively). Asterisks above bars denote significant differences (ANOVA; ^*p<0.05, ^***p<0.001) from vehicle-treated controls.

Figure 4

Mean error scores±SEM of all groups during each learning stage of reversal performance in Experiment 1: (a) perseveration (<50% correct), (b) learning (⩾50% correct). Perseveration stage: the main effects of group and reversal phase were significant (F_3,30=5.6, ^**p=0.004 and F_2,60=237.9, ^***p<0.001, respectively) as well as the group × reversal phase interaction (F_6,60=5.0, ^***p<0.001). Planned comparisons showed that animals treated with the two highest doses of M100907 (0.03 and 0.1 mg/kg) made significantly more perseverative errors than controls in reversal phase 1 (F_1,30=6.75, ^*p=0.014 and F_1,30=18.75, ^***p<0.001, respectively). Learning stage: there was no significant main effect of group (F_3,30=0.29, p=0.83), a significant main effect of reversal phase (F_2,60=17.34, ^***p<0.001), and no significant group × reversal phase interaction (F_6,60=0.80, p=0.80). Asterisks in white bars denote significant differences (ANOVA; ^*p<0.05, ^**p<0.1) from vehicle-treated controls.

In contrast, although SB 242084 did not alter significantly either the number of trials to reach criterion (Figure 5a) or the number of incorrect responses (Figure 6a) during retention (with or without reversal occurring) of the drug-free spatial discrimination or the previously acquired reversals, it facilitated reversal learning. Specifically, SB 242084 significantly decreased trials to criterion in reversal phase 1 at all doses (reversal 1—vehicle vs 0.1 mg/kg contrast: F_1,29=4.65, p=0.039; vehicle vs 0.3 mg/kg contrast: F_1,29=6.13, p=0.019; vehicle vs 1.0 mg/kg contrast: F_1,29=11.0, p=0.002; Figure 5b), while the two highest doses of the drug decreased both number of incorrect responses (reversal 1—vehicle vs 0.3 mg/kg contrast: F_1,29=5.71, p=0.024; vehicle vs 1.0 mg/kg contrast: F_1,29=8.37, p=0.007; Figure 6b) and perseverative errors (reversal 1—vehicle vs 0.3 mg/kg: F_1,29=5.17, p=0.03; vehicle vs 1.0 mg/kg: F_1,29=8.05, p=0.008; Figure 7). No differences were noted in learning errors (Figure 7).

Figure 5

Number of trials to criterion through (a) the retention phase (without reversal) and retention (initial) phase (preceding reversal) and (b) Reversal phase of Experiment 2. Data are presented as mean values±SEM (a) Retention phase (without reversal): there was no significant main effect of group (F_3,29=2.04, p=0.13), a significant main effect of retention phase (F_2,58=14.9, ^***p<0.001), and no significant group × retention phase interaction (F_6,58=1.07, p=0.39). Retention phase (preceding reversal): there was no significant main effect of group (F_3,29=1.64, p=0.20), a significant main effect of retention phase (F_2,58=34.47, ^***p<0.001), and no significant group × retention phase interaction (F_6,58=1.41, p=0.23). (b) Reversal phase: there was a significant main effect of reversal phase (F_2.58=96.81, ^***p<0.001) and a significant group × reversal phase interaction (F_6,58=3.39, ^**p=0.0063). Contrast testing demonstrated that SB 242084 significantly decreased trials to criterion compared with the vehicle group in reversal phase 1 at all doses (0.1 mg/kg: F_1,29=4.65, ^*p=0.039; 0.3 mg/kg: F_1,29=6.13, ^*p=0.019; 1.0 mg/kg: F_1,29=11.0, ^**p=0.002). Asterisks above bars denote significant differences (ANOVA; ^*p<0.05, ^**p<0.01) from vehicle-treated controls.

Figure 6

Incorrect responses of completed trials through (a) the retention phase (without reversal) and retention (initial) phase (preceding reversal) and (b) Reversal phase of Experiment 2. (a) Retention phase (without reversal): there were no significant differences between the groups in their ability to retain the drug-free acquired spatial discrimination or to retain previously acquired reversals (group: F_3,29=0.74, p=0.54; retention phase: F_2,58=21.81, ^***p<0.001; group × retention phase: F_6,58=1.38, p=0.24). Retention phase (preceding reversal): there was no significant main effect of group (F_3,29=1.05, p=0.38), a significant main effect of retention phase (F_2,58=12.34, ^***p<0.001), and no significant group × retention phase interaction (F_6,58=0.62, p=0.71). (b) Reversal phase: analysis yielded no significant main effect of group (F_3,29=2.4, p<0.09), a significant main effect of reversal phase (F_2,58=131.8, ^***p<0.001), and a significant reversal phase × group interaction (F_6,58=2.8, ^*p<0.019). Contrast testing revealed that rats treated with the two highest doses of SB 242084 (0.3 and 1.0 mg/kg) made significantly more incorrect responses than controls in reversal phase 1 (F_1,29=5.71, ^*p=0.024, and F_1,29=8.37, ^**p<0.007, respectively). Asterisks above bars denote significant differences (ANOVA; ^*p<0.05, ^**p<0.01) from vehicle-treated controls.

Figure 7

Mean error scores±SEM of all groups during each learning stage of reversal performance in Experiment 2: (a) perseveration (<50% correct), (b) learning (⩾50% correct). Perseveration stage: the main effect of reversal phase and the group × reversal phase interaction were significant (F_2,58=106.9, ^***p<0.001 and F_6,58=2.9, ^*p=0.014, respectively). Planned comparisons showed that the two highest doses of SB 242084 (0.3 and 1.0 mg/kg) significantly reduced perseverative errors compared with controls in reversal phase 1 (F_1,29=5.17, ^*p=0.03 and F_1,29=8.05, ^**p=0.008, respectively). Learning stage: there were no significant main effect of group (F_3,29=0.178, p=0.911), a significant main effect of reversal phase (F_2,58=5.06, ^**p=0.009), and no significant group × reversal phase interaction (F_6,58=0.613, p=0.72). Asterisks in white bars denote significant differences (ANOVA; ^*p<0.05, ^**p<0.01) from vehicle-treated controls.

Neither M100907- nor SB 242084-treated animals omitted more trials compared with vehicle-treated controls (Table 1) and there were no effects of either drug on the latencies to make a respond at any stage of the experiment (Table 2). Finally, no significant differences were noted between the groups in number of extra responses on the incorrect lever during correct scored trials (p-values>0.05; data not shown).

Table 1 Mean Values±SEM of Omissions during Retention (Collapsed Retention without Reversal and Retention (Initial Phase) Preceding Reversal) and Reversal Phase of Experiments 1 and 2

Table 2 Mean Values±SEM of Average Latencies to Respond (in Seconds) during Retention (Collapsed Retention without Reversal and Retention (Initial Phase) Preceding Reversal) and Reversal Phase of Experiments 1 and 2

It should be noted here that M100907 vehicle-treated rats performed better in Reversal 1 than SB 242084 vehicle-treated controls in the measures of incorrect responses and perseverative errors, though not in terms of number of trials. Current as well as previous (Boulougouris et al, 2007) work has defined the range of variation in control groups to be about 225±26 for incorrect responses and 57±17 for perseverative errors in Reversal 1. While not significantly outside the typical range of responding expected in this task (SB 242084 vs M100907; incorrect responses, 256±19.08 vs 196±15.26; perseverative errors, 74.45±6.51 vs 51.18±0.62), baseline differences even smaller than those reported here could influence drug response, a fact that should be kept in mind when interpreting these results.

DISCUSSION

We have demonstrated dissociable behavioral effects of the selective 5-HT_2A antagonist M100907 and the 5-HT_2C antagonist SB 242084 on serial spatial reversal learning. M100907 impaired initial reversal learning by increasing number of trials (highest dose only) and incorrect responses to criterion (two highest doses). This impairment, perseverative in nature, occurred in the absence of significant effects on retention of previous stimulus-reward contingencies. In contrast, SB 242084 improved reversal learning by decreasing the same measures (two highest doses). Our findings indicate that 5-HT_2A and 5-HT_2C receptors influence distinct aspects of behavioral flexibility. These dissociable effects were observed during Reversal 1 only. Failure of the drugs to affect Reversals 2 and 3 may be due to several reasons: (1) tolerance to the drug effects after repeated administration (effects of chronic administration of these drugs has not been reported); (2) Reversal 1 may be more drug sensitive, possibly due to its novelty; or (3) because it requires a large number of trials to criterion compared with Reversals 2 and 3; this reveals a possible learning set component specific to Reversal 1, which could be expected to benefit future reversals. It is noteworthy that OFC lesions (marmoset, Dias et al, 1996; rat, Boulougouris et al, 2007) also impaired the first reversal only. The single study reporting impaired first, second, and third reversals of odor discriminations after OFC lesions (McAlonan and Brown, 2003) employed a task where reversal sessions are not true serial reversals, as they occur with novel stimuli for each reversal.

In the present study, analysis of type of errors revealed that the opposing effects of the 5-HT_2A and 5-HT_2C receptor antagonists were specific to early Reversal 1 stages, affecting perseverative but not learning errors. Perseverative responding may have been modified as a result of changes either in prepotent response inhibition, or in the ability to detect contingency changes. We believe the perseverative deficit we noted reflects a selective influence on inhibitory response control: the alternative explanation, deficient detection of contingency changes, should have resulted in significant differences in (1) omissions and/or (2) number of incorrect responses until the stage where animals score their first correct trial during reversal (ie experience contingency shift). No such differences were observed.

Experiment 1: Effects of M100907 on Reversal Learning

M100907 affected neither rats' ability to perform a spatial discrimination learned prior to drug administration nor the late phases (ie ‘learning’ phases) of reversal learning. However, it significantly increased perseverative errors in the early stage of reversal learning. This finding is reminiscent of the effects of dietary tryptophan depletion in humans (Park et al, 1994) or selective (5,7-DHT) destruction of the ascending serotonergic projections in animals (Clarke et al, 2004, 2005, 2007). This perseverative deficit is likely to be mediated by orbitofrontal circuitry and its serotonergic innervation, which have previously been shown to play a critical role in response reversal. Specifically, we have previously demonstrated that bilateral excitotoxic lesions of the rat OFC (but not infralimbic or prelimbic cortex) impair reversal learning, a deficit manifested as increased perseverative responding to the previously reinforced lever (Boulougouris et al, 2007). Moreover, Clarke et al (2007) showed that selective 5-HT depletion of the marmoset OFC markedly impaired performance of a visual serial reversal learning task and this deficit was due to a failure to inhibit responding to the previously rewarded stimulus.

Previous studies have shown that 5-HT_2A receptor antagonism produces a functional enhancement of D₂ receptor antagonism under certain conditions. Selective blockade of 5-HT_2A receptors, for example, enhances the effect of D₂ receptor blockade on ventral midbrain DA cell firing (Olijslagers et al, 2004, 2005) and on limbic DA release (Bonaccorso et al, 2002; Liegeois et al, 2002). In reversal learning, there is evidence for an involvement of the DA system. Ridley et al (1981) demonstrated impaired reversal after the D₂ receptor antagonist haloperidol, while Lee et al (2007) showed that the selective D₂/D₃ receptor antagonist, raclopride, impairs reversal learning in monkeys. Furthermore, Floresco et al (2006) showed that administration of the D₂/D₃ subtype selective antagonist eticlopride potently impaired animals' ability to change their behavior in response to a conditional change of rule in a set-shifting task. Finally, selective blockade of D₂ receptor gene in knockout mice impaired reversal learning of an odor discrimination (Kruzich and Grandy, 2004).

It is worth noting that the results reported here reveal a dissociation between anticipatory responding in the 5CSRTT and reversal learning (ie impulsivity vs compulsivity). Specifically, studies utilizing the 5CSRTT have shown that M100907 decreases premature but not perseverative responses (Winstanley et al, 2003). These authors suggested that the latter measure does not represent perseverative responding at the aperture associated with reward but perseverative nose-poke activity at the array that is not punished. In another study (Carli et al, 2006), infusions of M100907 in the medial prefrontal cortex (mPFC) counteracted the loss of executive control (impulsivity increase induced by the competitive NMDA receptor antagonist 3-(R)-2-carboxypiperazin-4-propyl-1-propyl-1-phosphonic acid), while the selective 5-HT_1A receptor agonist 8-OH-DPAT decreased compulsive perseveration. Given the suggestion that perseveration in a response associated with reward delivery may be related to both impulsive and compulsive behavior, particularly when such an action is punished or not rewarded (Soubrié, 1986; Hollander and Rosen, 2000), these findings suggest that perseverative and premature responses in the 5CSRTT are differentially regulated by the 5-HT system. This lends support to the view that different aspects of impulsivity/compulsivity have distinct neurobiological substrates.

Experiment 2: Effects of SB 242084 on Reversal Learning

In contrast to M100907, SB 242084 actually improved serial spatial reversal learning by reducing the number of trials and incorrect responses to criterion in Reversal 1 compared to vehicle controls. Perseverative, but not learning, errors were also reduced.

5-HT_2C antagonism has previously been shown to mimic some of the effects of psychostimulant drugs such as D-amphetamine which increases DA release in the nucleus accumbens (Cole and Robbins, 1987, 1989): D-amphetamine causes a similar pattern of behavioral effects on the 5CSRTT as SB 242084, increasing the number of premature responses (Cole and Robbins, 1987; Harrison et al, 1997). Moreover, it has been shown to impair reversal-test performance (Ridley et al, 1998; Bensadoun et al, 2004; Idris et al, 2005). However, other studies have reported D-amphetamine facilitation of reversal learning in a two-choice simultaneous brightness discrimination (Weiner et al, 1986; Weiner and Feldon, 1986) and enhanced switching behavior (Evenden and Robbins, 1983; van den Bos and Cools, 1989). The present finding that SB 242084 decreased number of trials and incorrect (perseverative) responses may be consistent with these findings.

To our knowledge, this is the first demonstration of 5-HT_2C receptor involvement in reversal learning. Results on the effects of 5-HT_2C receptor agonists on compulsive behavior are equivocal. 5-HT_2C receptor activation induced ‘compulsive’ grooming (Graf et al, 2003; Graf, 2006) and directional persistence in spatial alternation (Tsaltas et al, 2005), while in other models 5-HT_2C agonists attenuated compulsive behavior (marble burying and schedule-induced polydipsia; Martin et al, 1998a). It should be noted that the anticompulsive effects of 5-HT_2C agonists have been attributed to their sedative effects (Kennett et al, 2000). On the other hand, blockade of 5-HT_2C receptors increased compulsive drinking in the polydipsia model (Martin et al, 2002), whereas E Tsaltas et al (unpublished observations) have shown that SB 242084 protected against meta-chlorophenylpiperazine (mCPP; a nonselective serotonin agonist)-induced directional persistence in the spatial alternation model of OCD. Finally, systemic administration of the 5-HT_2C receptor antagonist RS 10221 selectively decreased ‘surplus’ lever-pressing in the signal attenuation model (Flaisher-Grinberg et al, unpublished observations).

Effects of 5-HT_2C receptor antagonism have also been reported to enhance the stimulant effects (eg suppression of motivated behavior, locomotor activity, etc) of several drugs of abuse (phencyclidine, 3,4-methylenedioxymethamphetamine (MDMA); Hutson et al, 2000; Fletcher et al, 2001, 2002a, 2002b). Given that addiction and compulsivity seem to share underlying neural substrates such as the OFC and the DA system (Stein et al, 1995; Jentsch and Taylor, 1999; Everitt and Robbins, 2005; Kalivas and Volkow, 2005), the present finding that SB 242084 reduced perseverative responding suggests that 5-HT_2C receptors of distinct brain areas may be involved in compulsivity vs drug addiction. Accumulating evidence attributes the proaddictive effects of 5-HT_2C antagonists to increase of burst firing of dopaminergic neurons in the ventral tegmental area (VTA), leading to increased release of DA in the nucleus accumbens (Millan et al, 1998; Di Matteo et al, 1999, 2000a, 2000b, 2001, 2002; Gobert et al, 2000; Di Giovanni et al, 2001; Higgins and Fletcher, 2003). The finding that SB 242084 reduced perseverative responding in spatial reversal learning, a task dependent on the OFC (Boulougouris et al, 2007), suggests that this facilitatory effect of 5-HT_2C antagonists is possibly mediated by the OFC.

Pharmacological Specificity of the Drugs and 5-HT/DA Neurotransmission

5-HT_2C receptors are located in a variety of forebrain structures, including the neocortex, amygdala, hippocampus, dorsal, and ventral (including nucleus accumbens) striatal regions, as well as in monoaminergic cell body-rich areas such as the locus coeruleus, substantia nigra, and VTA (Pompeiano et al, 1994; Abramowski et al, 1995; Eberle-Wang et al, 1997). Eberle-Wang et al (1997) demonstrated the presence of 5-HT_2C mRNA within inhibitory GABAergic interneurons making direct synaptic contact with dopaminergic cell bodies in both the VTA and substantia nigra. The 5-HT_2A receptors are particularly prominent in cortical areas but are also found in DA-rich areas such as the striatum, substantia nigra, and VTA (Pompeiano et al, 1994; Lopez-Gimenez et al, 1997; Doherty and Pickel, 2000).

It has been shown that the 5-HT₂ receptor subtypes are differentially activated by 5-HT in vivo. M100907 does not influence the spontaneous firing rate of dopaminergic neurons or alter basal levels of DA or norepinephrine (noradrenaline, NA) release (Kehne et al, 1996), but attenuates amphetamine-induced hyperactivity (Sorensen et al, 1993) and amphetamine or DOI or MDMA-induced DA release (Schmidt et al, 1994; Gobert and Millan, 1999; Porras et al, 2002). In contrast, administration of 5-HT_2C receptor antagonists, including SB 242084, increases DA and NA release (Millan et al, 1998; Di Matteo et al, 2000a; Gobert et al, 2000) as well as VTA cell firing in the nucleus accumbens (Di Matteo et al, 1999). Moreover, SB 242084 produces behavioral effects in 5-HT-depleted animals (Winstanley et al, 2004), indicating that the 5-HT_2C receptors are tonically activated and that 5-HT_2C receptors are likely to be active under conditions of low-5-HT tone. Alternatively, these data may suggest that SB 242084 effects are not due to its actions at 5-HT receptors. However, SB 242084 is a high-affinity antagonist for the 5-HT_2C receptor (pK_i 9.0), 100-fold, 158-fold selectivity over the 5-HT_2B and 5-HT_2A receptors, respectively, and also has over 100-fold selectivity for the 5-HT_2C receptor over a range of other serotonergic, dopaminergic, and adrenergic receptors (Kennett et al, 1997). It seems unlikely that such marked behavioral effects known to be sensitive to manipulations of the 5-HT system are caused through the drug's actions at non-serotonergic receptors. Moreover, there is a debate whether SB 242084 acts as an inverse agonist at 5-HT_2C receptors rather than as a neutral antagonist (Barker et al, 1994), but there is no evidence to date supporting this possibility. In contrast, M100907 has been shown to exert its behavioral effects (reduced hyperactivity after NMDA receptor antagonism) under conditions of increased 5-HT release, while low-5-HT tone abolishes these effects (Martin et al, 1998b; Ceglia et al, 2004). This differentiation between 5-HT_2A and 5-HT_2C receptors may be attributed to the lower affinity of M100907 for the 5-HT_2A receptor than that of SB 242084 for the 5-HT_2C receptor, or to differences in selectivity of 5-HT for 5-HT_2C over 5-HT_2A receptors. However, these explanations seem unlikely (Winstanley et al, 2004) as the pK_i of M100907 for the 5-HT_2A receptor is 9.4 and the pK_i of SB 242084 for the 5-HT_2C receptor is 9.0 (Barnes and Sharp, 1999).

Obsessive-Compulsive Disorder and Reversal Learning

The present findings may be relevant to various neuropsychiatric disorders where inflexible behavior is a feature. Although OCD patients are not markedly impaired on simple reversal learning, they have impairments in other tasks sensitive to OFC function such as alternation learning, a task related to reversal learning (Freedman et al, 1998). They also show impairments on laboratory tests of frontal lobe function involving response shifting and inhibitory processing that correlate with the severity of their symptoms (Veale et al, 1996; Rosenberg et al, 1997; Schmidtke et al, 1998; Hollander and Rosen, 2000). The serotonergic system is also implicated in OCD, for example, via the therapeutic effects of specific serotonin reuptake inhibitors (SSRIs) (Baumgarten and Grozdanovic, 1998; El Mansari and Blier, 2006). Further investigation has implicated 5-HT₂ receptor families in the pathophysiology of OCD and in the mediation of the antiobsessive effects of SRIs. Treatment with psychedelic drugs possessing potent 5-HT_2A/2C agonist action properties appears to have favorable results on OCD patients (Moreno and Delgado, 1997; Delgado and Moreno 1998a, 1998b; Delgado, 2000), and 5-HT_2C receptor antagonism has been suggested to play a role in the generation of obsessive-compulsive symptoms in patients with comorbid psychiatric disorder, although this effect was not reported in patients suffering from primary/pure OCD (Khullar et al, 2001; see Sareen et al, 2004 for review). Furthermore, the 5-HT₂ antagonist ritanserin reversed the therapeutic effect of fluvoxamine (Erzegovesi et al, 1992), while studies which assessed the behavioral response to mCPP following chronic treatment with SSRIs have shown attenuated response to mCPP suggesting that chronic treatment with SSRIs leads to desensitization of 5-HT_2C receptors (Kennedy et al, 1993; Kennett et al, 1994; Maj et al, 1996; Yamauchi et al, 2004). This latter hypothesis has been strengthened from reports on 5-HT_2C downregulation following chronic treatment with SSRIs (van Oekelen et al, 2003; Serretti et al, 2004). Based on this line of evidence, the results presented here may be relevant to the pathophysiology of OCD, suggesting a potential role for 5-HT_2A and 5-HT_2C receptors when considering possible treatment strategies for this disorder.