Action discovery in the basal ganglia through reinforcement of spike timing-dependent plasticity

Abstract

Action discovery is the cognitive process of learning new actions to achieve desirable outcomes in the environment, through the formation of action-outcome associations. Multiple lines of evidence suggest that the basal ganglia, an evolutionarily ancient set of nuclei in the central nervous system, are critical to action discovery in the mammalian brain. Changes in the strength of synaptic connections between areas of the cerebral cortex that generate motor activity, and the striatum, the principal input nucleus of the basal ganglia, are thought to be principally involved in action discovery. Such plasticity is formalised in the rules of spike timing-dependent plasticity (STDP), in which the precise timing and order of pre and postsynaptic neural activity has been proposed – based largely on in vitro studies – as a generic mechanism to modify synaptic weights. To learn action-outcome associations for action discovery requires additional sensory and reward-related information about the outcome. Moreover, these reinforcement signals can occur at far greater timescales than the millisecond timing relevant to STDP, and are required to ‘work backwards’ to modulate the prior action- related activity. If STDP and reinforcement can interact to implement action discovery in vivo via the basal ganglia is the central question of this thesis. First, a novel, joystick-based behavioural task of action discovery in the rat was developed. In the switching phase of the task rats were continuously forced to discover new actions required to elicit the reward through trial and error. It was demonstrated that the conjunction of a light followed one second later by BSR was able to support learning in this complex task. Furthermore, omission of either of these reinforcement components impaired behavioural performance, suggesting that they are important to action discovery. The putative cellular mechanisms of action discovery with these motor and reinforcement signals were investigated in electrophysiological experiments in vivo, in anesthetised rats. With STDP protocols, it was demonstrated that when presynaptic cortical activity either preceded (‘pre-post’) or followed (‘post-pre’) postsynaptic striatal neuron activity, potentiation of corticostriatal synapses was unable to be induced in vivo, unlike in vitro experiments. However, when STDP pairings were associated with a behaviourally-relevant reinforcement set – the same as in the joystick task – long-lasting potentiation was induced with the pre-post protocol, and robust depression with the post-pre protocol. Thus bidirectional plasticity was found when delayed reinforcement signals interacted with different STDP pairings, in which the only change was whether the presynaptic activation arrived 10 ms before or 10 ms after the postsynaptic activity. Moreover, the conditioned light stimulus was required at one second from the STDP pairings for potentiation, and depression was induced when BSR was applied alone at two seconds from the pairings. Pharmacological manipulations indicated that this reward-modulated potentiation relied on both dopamine D1 and adenosine A2A postsynaptic signalling in the striatum. These findings indicate that corticostriatal STDP in vivo is only Hebbian when associated with appropriately timed reinforcement, and also highlight the potentially critical role that short-latency sensory signals can play in plasticity. They also provide a potential cellular mechanism for action discovery via basal ganglia circuits.