Global Journal of Life Sciences and Biological Research
Open Access

The Neuronal Mechanism Underlying the Influence of Time and Risk on Cost-Benefit Evaluation in Decision Making

Kaizhen Peng and Xiaomin Zhang^{* *}Correspondence: Xiaomin Zhang, Department of Physiology, School of Basic Medicine, Kunming Medical University, Kunming, China, Tel: 15887181609, Email:

Author info »

Introduction

An important issue of the cost-benefit decision making study is the coding of benefit, which is usually caused by confronting a positive outcome, a reward, following a particular choice behavior. Decades of efforts have been made to uncover the neuronal mechanism relates to the subjective coding of this reward. The DA signal used to be regarded as representing the “hedonic impact” induced by positive outcome including preferred food and drink, as well as addictive drugs, and in this way the subjective values of these rewarding items are reflected [1-6]. Later it is proved that the midbrain DA is involved in coding Reward Predicting Error (RPE), which stands for the difference between the predicted and the actual reward [7-11]. This theory suggests that before a decision, individuals make predictions to the potential value of the outcome of each option, and this prediction is generated based on their past experience about the same or similar situation. And when an outcome that better than the prediction is achieved, it plays the role as a “positive reinforcer” that could motivate one to repeat the same behavior in order to get the “positive reinforcer” again, therefore every new round of the choice behavior and its consequence updates the experience to provide information for future decisions [12,13]. This process is the so-called reinforcement learning which provides a basis for adaptation and is vital during species evolution. However, despite the decades of efforts, a precise relationship between dopaminergic electrical signal and the quantity, as well as quality of reward is still not fully illustrated yet [14,15]. But because of the clear relationship between DA and RPE, the DA system is still in the center of the decision-making process.

Most dopaminergic neurons located in the midbrain, including Substantia Nigra pars compacta (SNc) and ventral tegmental area (VTA) [16,17]. The electrical activity of a dopaminergic neuron contains two main components. One of them is the tonic activity which is a comparatively irregular low-frequency impulse and serves as a pacemaker, while the other is the phasic activity which is a regulated high-frequency burst firing responding to a particular event [18-20]. For instance, the phasic activity increases when animals receive unpredicted stimulus, mostly positive but also negative or salient stimulus [11,18,21-24]. Furthermore, it is recently found that different subcomponent of phasic firing may represent different state of the choice and is tightly regulated temporally and spatially [4,11,18,25,26].

A large amount of evidence has shown the similarity between midbrain dopamine mediated reward prediction error and the subjective utility of economic decision [27-30]. Particularly, the dopamine RPE signal satisfies several criterions of individual’s preference defined by economic utility theory, such as completeness and transitivity [30]. The dopamine RPE signal hence could serve as a bridge between the two research areas of neuroscience and economics in studying reward oriented and goal-directed behavior [31]. And furthermore, based on this correlation, research began to focus on identifying and classifying factors that could influence the subjective valuation of the reward [29-32].

Literature Review

The neuronal coding of factors that influence cost-benefit evaluation

In the past few decades accumulating evidence is beginning to uncover the neuronal mechanism underlying cost-benefit decision making [33-35]. Several important factors have been shown to influence the reward-oriented choice behavior, including quantity and quality of the reward, temporal delay before the delivery of the reward, uncertainty of receiving the reward, and the physical or mental effort required to obtain the reward [29,35-41]. When animals need to choose among multiple options, the quantity and quality of the expected reward of each option naturally of exerts predominant influence [42]. Meanwhile, several other factors can also modulate the choosing behavior. One factor is the temporal interval between the operant behavior and the reward delivery since it has been proved that a temporal delay could reduce the probability of animals choosing that option [43,44]. A second factor is the uncertainty of receiving the reward which is like the real world that most of the choices are made under a certain level of uncertainty. In majority cases, animals prefer the option with higher probability (more certain) [45], and this type of behavior is defined as risk-aversion or uncertainty aversion depends on whether the probability is known (risk) or unknown (uncertainty) [46-48]. And a third factor frequently discussed is the effort required for approaching and obtaining the reward. According to the current theories, the delay and uncertainty are regarded as parallel to cost [29], or are considered to be one member of cost, together with effort, to modify the subjective coding of the reward value [49-51]. However, from the perspective of the consumer choice theory of economics, cost should not be regarded as a determinant of the utility of the reward, but instead should be evaluated against the potential utility brought about by reward. It is the integrated evaluation of whether the utility of the reward worth the cost that finally determines the choice. Such discrepancy arises when the relationship among risk, time, effort and reward are investigated in different research area. In this work, we suggest that, according to the existing literatures investigating the neuronal mechanisms relate to these three factors, the effort is processed as a cost and evaluated against benefit coding, while risk and temporal delay influence the neuronal coding of both cost and benefit.

The influence of time on cost-benefit decision making

The influence of temporal delay on cost-benefit decision making: One problem that is usually confronted by mankind and animals is to make decision among options containing different temporal delays. Sometimes individual should consider giving up immediate reward to pursue long-term goal, such as whether to take daily exercise (giving up rest) for lifelong health, and whether to reduce the intake of sweets to prevent the gain of weight in future (to pursue health). In these situations, one needs to make a comparison between instant and remote benefit. Usually immediate reward is preferred even though it is lower in quality and quantity than the remote reward [52-56]. It is explained in the way that the subjective utility of the reward is discounting over time, and individual then chooses the option with the higher “time discounted value”[57-59]. And furthermore, because individuals are inclined to choose the immediate reward, therefore in order to pursue the long-term goal to achieve an overall higher benefit they need to be able to control their impulsiveness [60].

The fMRI scanning of human subjects found that multiple brain regions of the limbic system, including ventral striatum, media Orbitofrontal Cortex (mOFC), Posterior Cingulate Cortex (PCC), show increased activity when people choose an immediate reward [61,62]. However, the neuronal activities of the above brain regions are sensitive to both the reward quality and temporal delay [63]. More evidence comes from the in vivo single unit neuronal recording of monkeys. Studies show a positive relationship between the neuronal activity in striatum and timediscounted subjective value of the reward, while there is a reduced activity of the neuron in lateral cortex nucleus when the temporal duration increase [64]. And in consistent with other studies the neuronal activity was sensitive to both the reward magnitude and the duration of temporal delay [65]. This highly shared brain region and the similar neuron response between reward value coding and temporal delay duration seems to suggest an influence of temporal delay on the subjective coding of the value reward instead of a tradeoff between time and reward temporal delay is regarded as a cost.

Rodents are also sensitive to temporal delay of the reward. Evenden et al designed a behavior task to test the time discounting rate of rats [66], and pharmacological experiments were performed with this behavior task. In the task, rats are trained to choose between an immediate small food and a delayed large food (5 folds to the alternative) and the choices of rats are shown by lever pressing. Using this behavior task, a time discounting curve can be drawn by gradually increase the temporal delay duration of the larger food (10, 20, 40, 60 seconds) and record the choice of the rats accordingly. They find that intraperitoneal injection of amphetamine reduces the probability of choosing the larger but delayed reward, and the GABA receptor agonist diazepam significantly increases the choices of delayed food when the waiting time was 10 or 20 seconds [66]. The in vivo single unit recording of the dopamine neurons in VTA showed increases activity when rats chose immediate small foods [67]. And D2 receptor knockdown in VTA enhanced the immediate choice of rats [68] which suggests that the manipulation causes a stronger temporal discounting effect on the subjective coding of remote reward. NAc is also involved in time related tasks. Lesion to NAc causes an “impulsive” choice that rats prefer immediate reward [69-71]. However, when the two options show the same waiting time rats still chose larger rewards which suggests an unbiased preference to food itself. Another study shows that rats with lesion in NAc are less sensitive to both reward magnitude and temporal delay [72]. Furthermore, the availability of D2/3 receptors in Nucleus Accumbens core (NAcc) is inversely related to the impulsive behavior, that lower D2/3 receptor level predicts higher impulsivity [73]. These studies indicated that NAc may control the temporal discounting effect on the remote reward, and similar with VTA, lesion to NAc also causes a sharp decrease of subjective value of the remote reward [74]. The midbrain DA system is known to represent the expectation of the reward, therefore based on the above-mentioned experimental results it seems that the observed DA response change during a temporal discounting task may only reflect discounted reward but not “discounting” itself. Because if the subjective coding of remote reward diminished after dis-functioning of VTA or NAc, rats may estimate the option less attractive compare to the instant reward even if the delay duration is not changed.

Besides limbic midbrain regions, Winstanley et al found that OFC and BLA are also involved in time discounting choices and these two brain regions seems to play opposite roles. Lesion to BLA causes a preference towards the immediate reward, while rats with a lesion to OFC are more willing to wait for the larger reward [75]. The single unit recordings of neurons in OFC of both non-human primates and rats show a stronger activity when the subject receives a cue predicting a short waiting time [76,77]. The amygdala neurons also shows stronger activity to the cue that predicts longer waiting time [78,79]. Dorsolateral Prefrontal Cortex (DLPFC) is another relevant brain region, but different from OFC, the DLPFC neurons responding not only to delay but also to reward, therefore it is hypothesized to encode the discounted reward value while OFC more closely related to the “temporal delay” itself [80]. And since the prefrontal region of cortex plays important roles in self-control [81-83], it has been suggested that the neuronal signal of OFC represents the ability to inhibit the impulsivity towards instant rewards [84].

The fMRI study of human subjects showed that the reward devaluation caused by temporal discounting is also related to ACC and mPFC, which are two brain regions less involved in risk related tasks [85,86]. Both ACC and mPFC play important roles in cognitive control, and ACC is suggested to be responsible for conflict detection, so the involvement of these two brain regions suggests a high requirement of cognitive control during temporal discounting tasks. This activation of the frontal areas that in charges of cognitive control is accordance with the need of overcoming one’s impatience in order to pursue larger long-term benefit in temporal related choices.

Therefore, it seems that time has two types of impact on costbenefit evaluation. One is that the delayed supply of reward will cause a reduction of its subjective value, and another is that choosing to wait for the remote and large reward requires cognitive control over one’s own impatience which should be regarded as a kind of mental cost, and accordingly different brain regions are motivated. The mental cost, although has not been clearly defined, is closely related to cognitive control and depends on neuronal activity of PFC [87]. However interestingly, different from the fMRI results of human subjects, lesions to ACC and mPFC of rats have no effect on the choices with different temporal discount [69]. Such difference between human and non-human subjects is not seen in the neuronal processing of effort or risk, which indicates that it is a special feature of time related choice, and also indicates a higher requirement of cognitive control in time related tasks compared to the other two. In summary, as has been mentioned, the temporal delay may bias the cost-benefit evaluation in two ways, the first of which is to directly diminish the subjective value of the outcome, and the second is to increase the mental cost, and as a result it will change the balance between cost and benefit. Furthermore, the lack of impact on temporal discounting tasks after lesion to mPFC or ACC might suggest that for rodents, mental cost is not strongly required for the time related tasks, which indicates that the evaluation of cost against benefit is different between human and non-human animals.

The influence of risk on cost-benefit decision making

Another factor that could influence the decision making is the uncertainty of the outcome. Actually, most choices obtain a certain level of uncertainty. For instance, when a person chooses to invest stock A or stock B, then either choice may cause gain or loss, in spite of different probability.

It has been proved that animals are risk-sensitive [88,89]. The early studies of this issue focused on foraging behavior and found that birds make riskier choice before migration period. Caraco et al provides the first conception of risk-sensitivity theory based on experimental examination of the foraging behavior of the yellow-eyed junco bird (a kind of migratory bird) [90]. In their experiments, birds need to choose between two food supply patterns with the same mean reward size: three seeds each time for certain versus six seeds or nothing with equal probability. They find that the choices of birds are affected by the environmental temperature. When the temperature is low, three seeds are not enough to maintain the body temperature and then birds are prone to choose more risky option in order to have a chance of getting six seeds and survive. However, when the temperature raises up to 19⁰C and three seeds are enough for survival, birds are risk aversive and prone to choose safe option [90]. They conclude in their optimal foraging theory that the bird’s preference for foraging strategy is in accordance with the environmental stochasticity. Consistently, Moore et al. suggests that migrants need to reach a higher body weight in order to survive the migration [91,92]. These studies prove that migratory birds are risk sensitive, and they adjust decision strategies to adapt to the changing environment.

The later studies further investigate the neuronal mechanisms involved in choice with uncertainty, and accumulating evidence suggests the involvement of dopamine signal transmission [93- 95]. In a rodent version of Iowa Gambling Task (IGT), rats or mice show their choice by hopping into one of the goal arms with foods of different size and probability [96]. Using this behavior task Young et al find that the Dopamine Transporter (DAT) knock-out mice more frequently choose arm A and B which suggests that they are risk-preferred [97].

But the role of dopamine in risk related decision seems to be complex. St. Onge et al. used a lever-pressing behavior task to investigate the risk attitude of rats. In their experiment, rats showed their choice between two options by pressing different levers. One lever delivers a certain reward of one food pallet while the other lever delivers a larger reward but the probability of receiving it decreases across the four trial blocks from 100% to 12.5% [98]. They find that systematically enhance the DA signaling by amphetamine or D1R and D2R agonists increases the preference for the larger but risky choice, while inhibition of DA signaling results in risk aversion like behaviors. In another similar task, rats were required to choose between a small, safe reward and a large reward possibly companied with foot shock [99]. Different from the former mentioned study, in this task, a similar level of amphetamine resulted in risk aversion [99]. The same group later found a non-uniformly distributed risk attitude among rats and according to which rats can be classified into three groups: riskseeking group, risk-neutral group and risk-aversive group, and the D1R mRNA level of the insular cortex of risk-seeking rats and the D2R mRNA level of the striatum of the risk-aversive rats are higher than the other two groups [100]. Moreover, they find the intraperitoneal injection of D2 receptor agonist bromocriptine increase the “safe” choice while D1R receptor agonist has no effect on risky choices [100]. Therefore, the dopamine signaling is important in risk related choice, but its role is still not clear yet. An explanation of this discrepancy is that different component of dopamine signals relates to different aspects of risky choices. Christopher et al first identifies that the phasic activity of dopaminergic neuron varies monotonically with the probability of the reward [101]. In their experiments, the monkey is given a reward after a particular visual signal. After extensive training the reward receiving no longer elicit a phasic dopamine signal since a solid conditional relationship between visual signal and reward has been generated. However, if the probability of the reward is modified to be less than 100%, the phasic response appears again. This electrical signal reaches maximal when the probability is 50% and it would not decay with the repetition of the test. Another explanation is that dopamine neurons of different brain regions may function differently. Recently, Zalocusky et al investigates the causal relationship between DA neuronal circuit and risk attitude for the first time by using optogenetics [101]. In their experiments, rats chose between a safe choice (50 μl sucrose) and a risky choice (25% of 170 μl sucrose and 75% 10 μl sucrose). In consistent with Setlow’s findings they also find a diversity of rats’ risk attitude. They further express channel rhodopsin on the D2R positive neurons of NAc and activates these neurons by blue light. As a result, the risky choice of the risk-seeking rat is inhibited after stimulation of this group of DA neurons while the choice of risk-aversive rat is not affected [101]. This study generates a causal relationship between a functional specified group of dopamine neurons and the risk aversive behavior. Other studies focusing on the circuit level mechanism of risk attitude also find that the dopaminergic neurons of VTA and its downstream targets are related to perception of uncertainty [102-104]. These studies suggest that similar to temporal delay, uncertainty also influences the subjective coding of the reward and in this way modifies the choice behavior.

However other than influencing the valuation of reward, uncertainty can also influence the decision making in other ways. For instance, according to the current knowledge, most individuals, across species, are risk aversive, which means that they prefer safe choice than risky choice, and this is explained by loss aversion, which is an important feature in decision making under uncertainty from the perspective of economics [105]. It refers to the observation that people intend to pay a larger amount of money to avoid a potential loss. The main brain region that closely related to loss aversion is Amygdala, which as mentioned above, is important for emotion processing, especially fear, threat and anxiety [106-108], which are all sensations felt when people experience loss or a potential loss [109]. Lesion to amygdala induced a higher probability of engaging in gambles regardless of potential loss [110]. Consistently, fMRI studies show that the activity of amygdala is more strongly coupled to loss related decision [111,112]. Other studies find that Amygdala is also involved in decisions with uncertainty, even without a potential loss [113,114]. However, it could be interpreted as that the feature of “uncertainty” itself is enough to generate anxiety and stress. Recently, a circuit level study show that the ACC to BLA monosynaptic glutamatergic projection controls the innate fear response of rats [115]. In this study, the innate fear is induced by predator odor, and foot shock is used as a conditioned fear control. It is found that optogenetic inactivation of ACC to BLA projection enhanced freezing response to innate fear but not conditioned fear [115]. It would be interesting to evaluate the influence of this manipulation on choice behavior with uncertainty, because if the aversion to loss is rooted in innate fear, the choice behavior with uncertainty should be influenced by the optogenetic manipulation of this neuronal circuit.

In brief summary, according to the current evidence, the factor of uncertainty not only influences evaluation of reward, but also cost. According to expected utility theory, the subjective value of the reward equals to the objective value multiplied by the probability, while prospect theory suggests that the subjective reward is valued by weighting objective value according to its probability. In the brain, the activity change of midbrain dopamine neurons according to uncertainty level may represent this probability discounting effect. But as mentioned above, each individual has its own risk attitude which varies largely, thus for a risk aversive person or animal, to overcome the natural inclination of choosing the safe choice and to diminish the anxiety induced by the uncertainty require a certain level of cognitive control, and the energy used for this purpose should be regarded as mental cost. Therefore, similar to the factor of time, uncertainty also influences the cost-benefit evaluation in two ways that it exerts impact on both reward coding and effort evaluation.

Discussion

In this paper, we made a review of the neuronal process underlying some important factors that could influence the decision making, including time and risk. The relationship among these two factors has been discussed for decades but still without consensus yet. Multiple brain regions, including VTA, striatum, frontal cortex, and especially dopaminergic signaling among these regions are related to all the three factors and obviously influence decision making. DA receptor agonist, antagonist or DAT inhibitor is able to modify behavior either in the time or risk related choice tasks. The highly overlapping of the function of these different brain regions indicates a functional diversification of neurons in each brain region, and also it indicates that the interconnection among subgroup of neurons and the directed information flow within the neuronal circuit might play important roles. For instance, in a time related task, one need to sense and measure the temporal delay, overcome its own impatience, and evaluate the size of the objective reward and the discounted reward, and each of them may be processed by different neurons. In addition, analyzing time and risk have some shared features, especially that they all require prediction and evaluation of the outcome of certain operant behavior which is consistent with the findings that VTA activity is detected in either behavior tasks. Currently, the cost-benefit evaluation is believed to be a useful framework in analyzing decision making. According to the existing evidence, we suggest a neuronal model under this framework (Figure 1).

Figure 1: Hypothetical model of factors influencing cost-benefit decision making processes. In this model, benefit is evaluated against the cost to determine whether an approaching action would be initiated. The cost is mainly decided by the physical effort required for obtaining the reward, as well as the mental effort caused by cognitive control in the time and risk involve choices. And the benefit is determined by the property of the reward and influenced by factors including risk and delay. Therefore in this model the “effort”, “time” and “risk” are not regarded to exert same influence on cost-benefit decision making.

Conclusion

Under this framework, the behavior is chosen after evaluating the required cost against potential benefit. The three factors, time, risk and effort influence cost and benefit in different ways. In time related choice, the subjective coding of the potential reward is discounted by the temporal delay, and meantime in order to overcome the natural preference towards immediate reward, certain level of mental effort is required so time also influence the cost evaluation. Similarly, the uncertainty diminishes the valuation of the outcome and makes it a less preferred choice. However, if an overall gain can only be achieved through making riskier choice, then a risk aversive person must overcome the tendency towards safe choice, which also requires cognitive control and should be regarded as mental cost. Therefore, we suggest that in a cost-benefit analysis, the temporal delay and uncertainty should not be only regarded as constituents of cost or be treated as parallel to cost, instead, they have impacts on both cost and benefit evaluation. And then a comparison of cost and benefit is carried out to decide whether to initiate an approaching action.

Author Contributions

Conceptualization, P.K.Z and Z.X.M.; writing—original draft preparation, Z.X.M.; writing—review and editing, P.K.Z.; funding acquisition, Z.X.M.

Funding

This work was supported by the National Natural Science Foundation of China (Grant No. 31960168).

Conflict of Interest

The authors declare no conflict of interest.

References

1. Bardo MT. Neuropharmacological mechanisms of drug reward: beyond dopamine in the nucleus accumbens. Crit Rev Neurobiol. 2012; 12 (1-2):37-68.

   [Crossref][Google scholar][PubMed]

2. Bechara A, Harrington F, Nader K, van der Kooy D. Neurobiology of motivation: double dissociation of two motivational mechanisms mediating opiate reward in drug-naive versus drug-dependent animals. Behav Neurosci. 1992; 106 (5):798-807.

   [Crossref][Google scholar][PubMed]

3. Ettenberg A. Dopamine, neuroleptics and reinforced behavior. Neurosci Biobehav Rev. 1989; 13(2-3):105-111.

   [Crossref][Google scholar][PubMed]

4. Schultz W, Apicella P, Ljungberg T. Responses of monkey dopamine neurons to reward and conditioned stimuli during successive steps of learning a delayed response task. J Neurosci. 1993; 13(3): 900-913.

   [Crossref][Google scholar][PubMed]

5. Koob GF. Hedonic valence, dopamine and motivation. Mol Psychiatry. 1996; 1(3):186-189.           

   [Google scholar][PubMed]

6. Carrián OA, Stamelou M, Rodríguez EM, Gonzlez MM, Pöppel E. Dopaminergic reward system: A short integrative review. Int Arch Med. 2010; 3(1):24.

   [Crossref][Google scholar][PubMed]

7. Schultz W, Dayan P, Montague PR. A neural substrate of prediction and reward. Sci. 1997; 275(5306):1593-1599.

   [Crossref][Google scholar][PubMed]

8. Uchida MW, Eshel N, Uchida N. Neural Circuitry of Reward Prediction Error. Annu Rev Neurosci .2017; 40(1):373-394.

   [Crossref][Google scholar][PubMed]

9. Bayer HM, Glimcher PW. Midbrain dopamine neurons encode a quantitative reward prediction error signal. Neuron. 2005; 47(1):129-141.

   [Crossref][Google scholar][PubMed]

10. Steinberg EE , Keiflin R, Boivin JR, Witten IB, Deisseroth K, Janak PH. A causal link between prediction errors, dopamine neurons and learning. Nat Neurosci. 2013; 16(7):966-973.

    [Crossref][Google scholar][PubMed]

11. Schultz W. Predictive Reward Signal of Dopamine Neurons. J Neurophysiol. 1998; 80(1):1-27.

    [Crossref][Google scholar][PubMed]

12. Premack D. Toward empirical behavior laws: I. Positive reinforcement. Psychol Rev.1959; 66(4):219-33.

    [Crossref][Google scholar][PubMed]

13. Salamone JD, CorreaM. Motivational views of reinforcement: Implications for understanding the behavioral functions of nucleus accumbens dopamine. Behavn Brain Res. 2002; 13(1-2):3-25.

    [Crossref][Google scholar][PubMed]

14. Berridge KC, Robinson TE. What is the role of dopamine in reward: Hedonic impact, reward learning, or incentive salience? Brain Res Rev. 1998; 28 (3):309-69.

    [Crossref][Google scholar][PubMed]

15. Shizgal P, Conover K. On the neural computation of utility. Curr Dir Psychol Sci. 1996; 5(2): 37-43.

    [Crossref][Google scholar]

16. Schultz W. Multiple dopamine functions at different time courses. Annu Rev Neurosci. 2007; 30(1):259-288.

    [Crossref][Google scholar][PubMed]

17. Gantz SC, Ford CP, Morikawa H, Williams JT. The Evolving understanding of dopamine neurons in the substantia nigra and ventral tegmental area. Annu Rev Physiol. 2017; 80(1):219-241.

    [Crossref][Google scholar][PubMed]

18. Schultz W. Dopamine reward prediction-error signalling: A two-component response. Nat Rev Neurosci. 2016;17(3):183-195.

    [Crossref][Google scholar][PubMed]

19. Schultz W. Electrophysiological correlates of reward processing in dopamine neurons. Decision Neuroscience: An Integrative Perspective. 2017; 21-31.

    [Crossref][Google scholar]

20. Stauffer WR, Lak A, Kobayashi S, Schultz W. Components and characteristics of the dopamine reward utility signal. J Comp Neurol.2016; 524(8): 1699-1711.

    [Crossref][Google scholar][PubMed]

21. Schultz W. Dopamine reward prediction error coding. Dialogues Clin Neurosci. 2016; 18(1):23-32.

    [Crossref][Google scholar][PubMed]

22. Bayer HM, Glimcher PW. Midbrain dopamine neurons encode a quantitative reward prediction error signal. Neuron. 2005; 47(1):129-141.

    [Crossref][Google scholar][PubMed]

23. Schultz W. Updating dopamine reward signals. Curr Opin Neurobiol. 2013; 23(2):229-238.

    [Crossref][Google scholar][PubMed]

24. Steinberg EE , Keiflin R, Boivin JR,  Witten IB, Deisseroth K, Janak PH. A causal link between prediction errors, dopamine neurons and learning. Nat Neurosci. 2013; 16(7):966-73.

    [Crossref][Google scholar][PubMed]

25. Nomoto K, Schultz W, Watanabe T, Sakagami, M. Temporally Extended Dopamine Responses to Perceptually Demanding Reward-Predictive Stimuli. J Neurosci. 2010; 30(32):10692-10702.

    [Crossref][Google scholar][PubMed]

26. Enomoto K, Matsumoto N, Nakai S, Satoh T, Sato TK, Ueda Y, et al.Dopamine neurons learn to encode the long-term value of multiple future rewards. Proc Natl Acad Sci. 2011; 108(37):15462-15467.

    [Crossref][Google scholar][PubMed]

27. Montague PR, Berns GS. Neural economics and the biological substrates of valuation. Neuron. 2002; 36(2):265-284.

    [Crossref][Google scholar][PubMed]

28. Rangel A, Camerer C, Montague PR. A framework for studying the neurobiology of value-based decision making. Nat Rev Neurosci. 2008; 9(7):545-56.

    [Crossref][Google scholar][PubMed]

29. Schioppa CP. Neurobiology of Economic Choice: A Good-Based Model. Annu Rev Neurosci. 2011; 34(1):333-59.

    [Crossref][Google scholar][PubMed]

30. Lak A, Stauffer WR, Schultz W. Dopamine prediction error responses integrate subjective value from different reward dimensions. Proc Natl Acad Sci. 2014; 111(6):2343-2348.

    [Crossref][Google scholar][PubMed]

31. Montague PR, Berns GS. Neural economics and the biological substrates of valuation. Neuron. 2002; 36(2):265-84.

    [Crossref][Google scholar][PubMed]

32. Kable JW, Glimcher PW. The neurobiology of decision: consensus and controversy. Neuron. 2009; 63(6):733-45.

    [Crossref][Google scholar][PubMed]

33. Basten U, Biele G, Heekeren HR, Fiebach CJ. How the brain integrates costs and benefits during decision making. Proc Natl Acad Sci. 2010; 107(50):21767-21772.

    [Crossref][Google scholar][PubMed]

34. Fischhoff B. The realities of risk-cost-benefit analysis. Sci. 2015; 350(6260):aaa6516          

    [Crossref][Google scholar][PubMed]

35. Floresco SB, Onge JRS, Sharifi SG, Winstanley CA. Cortico-limbic-striatal circuits subserving different forms of cost-benefit decision making. Cogn Affect Behav Neurosci. 2008; 8(4):375-389.

    [Crossref][Google scholar][PubMed]

36. Bailey MR, Simpson EH, Balsam PD. Neural substrates underlying effort, time, and risk-based decision making in motivated behavior. Neurobiol. Learn Mem. 2016; 133:233-256.

    [Crossref][Google scholar][PubMed]

37. Orsini CA, Moorman DE, Young JW, Setlow B, Floresco, SB. Neural mechanisms regulating different forms of risk-related decision-making: Insights from animal models. Neurosci Biobehav Rev. 2015; 58:147-167.

    [Crossref][Google scholar][PubMed]

38. Huettel SA, Stowe CJ, Gordon EM, Warner BT, Platt ML. Neural signatures of economic preferences for risk and ambiguity. Neuron. 2006; 49(5):765-775.

    [Crossref][Google scholar][PubMed]

39. Kuhnen CM, Knutson B. The neural basis of financial risk taking. Neuron.2005; 47(5):763-770.

    [Crossref][Google scholar][PubMed]

40. Peters J, Büchel C. The neural mechanisms of inter-temporal decision-making: Understanding variability. Trends Cogn Sci. 2011; 15(5):227-239.

    [Crossref][Google scholar][PubMed]

41. Ballard K, Knutson B. Dissociable neural representations of future reward magnitude and delay during temporal discounting. Neuroimage. 2009; 45(1):143-150.

    [Crossref][Google scholar][PubMed].

42. Plassmann H, Doherty JO, Rangel A. Orbitofrontal cortex encodes willingness to pay in everyday economic transactions. J Neurosci. 2007; 27(37):9984-9988.

    [Crossref][Google scholar][PubMed]

43. Mauk MD, Buonomano DV. The neural basis of temporal processing. Annu Rev Neurosci. 2004; 27:307-340.

    [Google scholar][PubMed]

44. Peters J, Büchel C. The neural mechanisms of inter-temporal decision-making:understanding variability. Trends Cogn Sci. 2011; 15(5):227-239.

    [Crossref][Google scholar][PubMed]

45. Peters J, Buchel C. Overlapping and distinct neural systems code for subjective value during intertemporal and risky decision making. J Neurosci. 2009; 29(50):15727-15734.

    [Crossref][Google scholar][PubMed]

46. Levy I, Snell J, Nelson AJ, Rustichini A, Glimcher PW. Neural representation of subjective value under risk and ambiguity. J Neurophysiol. 2010;103 (2):1036-1047.

    [Crossref][Google scholar][PubMed]

47. Bossaerts, P. Risk and risk prediction error signals in anterior insula. Brain Struct Funct. 2010; 214(5-6):645-653.

    [Crossref][Google scholar][PubMed]

48. McCoy AN, Platt ML. Risk-sensitive neurons in macaque posterior cingulate cortex. Nat Neurosci. 2005; 8(9):1220-1227.

    [Crossref][Google scholar][PubMed]

49. Kobayashi S, Schultz W. Influence of reward delays on responses of dopamine neurons. J Neurosci. 2008; 28 (31):7837-7846.

    [Crossref][Google scholar][PubMed]

50. Pippas LG, Tobler PN, Schultz W. Short-term temporal discounting of reward value in human ventral striatum. J Neurophysiol. 2009; 101(3):1507-1523.

    [Crossref][Google scholar][PubMed]

51. Schultz W. Subjective neuronal coding of reward: Temporal value discounting and risk. Eur J Neurosci. 2010; 31(12):2124-2135.

    [Crossref][Google scholar][PubMed]

52. Cardinal RN, Pennicott DR, Sugathapala CL, Robbins TW, Everitt BJ. Impulsive choice induced in rats by lesions of the nucleus accumbens core. Sci. 2001; 292(5526):2499-501.

    [Crossref][Google scholar][PubMed]

53. Kahneman D, Tversky A. Choices, values, and frames. Am Psychol. 1984; 39(4), 341–350.

    [Crossref][Google scholar]

54. Kalenscher T, Windmann S, Diekamp B, Rose J, Güntürkün O, Colombo M. Single units in the pigeon brain integrate reward amount and time-to-reward in an impulsive choice task. Curr Biol. 2005; 15(7):594-602.

    [Crossref][Google scholar][PubMed]

55. Mobini S, Body S, Ho MY, Bradshaw CM, Szabadi E, Deakin JFW. Effects of lesions of the orbitofrontal cortex on sensitivity to delayed and probabilistic reinforcement. Psychopharmacol Berl. 2002; 160(3):290-298.

    [Crossref][Google scholar][PubMed]

56. Winstanley CA.Contrasting roles of basolateral amygdala and orbitofrontal cortex in impulsive choice. J Neurosci Res. 2004; 24(20):4718-4722.

    [Crossref][Google scholar][PubMed]

57. Frederick S, Loewenstein G, O’donoghue T. Time discounting and time preference: a critical review. J Econ Lit. 2014;40(2):351-401.

    [Crossref][Google scholar]

58. Green L, Myerson J. A discounting framework for choice with delayed and probabilistic rewards. Psychol Bull.2004; 130(5):769-792.

    [Crossref][Google scholar][PubMed]

59. Kalenscher T, Pennartz CMA. Is a bird in the hand worth two in the future? The neuroeconomics of intertemporal decision-makingProg Neurobiol. 2008; 84(3):284-315.

    [Crossref][Google scholar][PubMed]

60. Salamone JD, Pardo M, Yohn SE, Cruz LL, SanMiguel N, Correa M. Mesolimbic dopamine and the regulation of motivated behavior. Curr Top Behav Neurosci. 2016; 27:231-257.

    [Crossref][Google scholar][PubMed]

61. McClure SM, Laibson DI, Loewenstein G, Cohen JD. Separate neural systems value immediate and delayed monetary rewards. Sci. 2004; 306(5695):503-507.

    [Crossref][Google scholar][PubMed]

62. McClure SM, Laibson DI, Cohen JD, Loewenstein G, Ericson KM. Time discounting for primary rewards. J Neurosci. 2007; 27(21):5796-804.

    [Crossref][Google scholar][PubMed]

63. Kable JW, Glimcher PW. The neural correlates of subjective value during intertemporal choice. Nat Neurosci. 2007; 10(12):1625-1633.

    [Crossref][Google scholar][PubMed]

64. Ballard K, Knutson B. Dissociable neural representations of future reward magnitude and delay during temporal discounting. Neuroimage. 2009; 45(1):143-150.

    [Crossref][Google scholar][PubMed]

65. Cai, X, Kim S, Lee D. Heterogeneous coding of temporally discounted values in the dorsal and ventral striatum during intertemporal choice. Neuron. 2011; 69(1):170-182.

    [Crossref][Google scholar][PubMed]

66. Evenden JL, Ryan CN. The pharmacology of impulsive behaviour in rats: The effects of drugs on response choice with varying delays of reinforcement. Psychopharmacol Berl. 1996; 128 (2):161-170.

    [Crossref][Google scholar][PubMed]

67. Roesch MR, Calu DJ, Schoenbaum G. Dopamine neurons encode the better option in rats deciding between differently delayed or sized rewards. Nat Neurosci. 2007; 10(12):1615-1624.

    [Crossref][Google scholar][PubMed]

68. Bernosky-Smith KA, Qiu YY, Feja M, Lee YB, Loughlin B, Li JX, et al. Ventral tegmental area D2 receptor knockdown enhances choice impulsivity in a delay-discounting task in rats. Behav Brain Res. 2018; 341:129-34.

    [Crossref] [Google Scholar] [PubMed]

69. Cardinal RN, Pennicott DR, Lakmali C, Sugathapala, Robbins TW, Everitt BJ. Impulsive choice induced in rats by lesions of the nucleus accumbens core. Sci. 2001; 292(5526):2499-2501.

    [Crossref] [Google Scholar] [PubMed]

70. Basar K, Sesia T, Groenewegen H, Steinbusch HW, Visser-Vandewalle V, Temel Y. Nucleus accumbens and impulsivity. Prog Neurobiol. 2010; 92(4):533-557.

    [Crossref]  [Google Scholar] [PubMed]

71. Bezzina G, Cheung TH, Asgari K, Hampson CL, Body S, Bradshaw CM, et al. Effects of quinolinic acid-induced lesions of the nucleus accumbens core on inter-temporal choice: a quantitative analysis. Psychopharmacol. 2007; 195(1):71-84.

    [Crossref]  [Google Scholar] [PubMed]

72. Steele CC, Peterson JR, Marshall AT, Stuebing SL, Kirkpatrick K. Nucleus accumbens core lesions induce sub-optimal choice and reduce sensitivity to magnitude and delay in impulsive choice tasks. Behav Brain Res. 2018; 339:28-38.

    [Crossref]  [Google Scholar]

73. Barlow RL, Gorges M, Wearn A, Niessen HG, Kassubek J, Dalley JW, et al. Ventral striatal D2/3 receptor availability is associated with impulsive choice behavior as well as limbic corticostriatal connectivity. Int J Neuropsychopharmacol. 2018; 21(7):705-715.

    [Crossref]  [Google Scholar] [PubMed]

74. Galtress T, Kirkpatrick K. The role of the nucleus accumbens core in impulsive choice, timing, and reward processing. Behav Neurosci. 2010; 124(1):26.

    [Crossref]  [Google Scholar] [PubMed]

75. Winstanley CA, Theobald DE, Cardinal RN, Robbins TW. Contrasting roles of basolateral amygdala and orbitofrontal cortex in impulsive choice. J Neurosci. 2004; 24(20):4718-22.

    [Crossref]  [Google Scholar] [PubMed]

76. Roesch MR, Olson CR. Neuronal activity in primate orbitofrontal cortex reflects the value of time. J Neurophysiol. 2005; 94(4):2457-2471.

    [Crossref]  [Google Scholar]

77. Roesch MR, Taylor AR, Schoenbaum G. Encoding of time-discounted rewards in orbitofrontal cortex is independent of value representation. Neuron. 2006; 51(4):509-520.

    [Crossref] [Google Scholar] [PubMed]

78. Bermudez MA, Göbel C, Schultz W. Sensitivity to temporal reward structure in amygdala neurons. Curr Biol. 2012; 22(19):1839-1844.

    [Crossref]  [Google Scholar] [PubMed]

79. Belova MA, Paton JJ, Morrison SE, Salzman CD. Expectation modulates neural responses to pleasant and aversive stimuli in primate amygdala. Neuron. 2007; 55(6):970-984.

    [Crossref] [Google Scholar] [PubMed]

80. Kim S, Hwang J, Lee D. Prefrontal coding of temporally discounted values during intertemporal choice. Neuron. 2008; 59(1):161-172.

    [Crossref]  [Google Scholar] [PubMed]

81. Ridderinkhof KR, Ullsperger M, Crone EA, Nieuwenhuis S. The role of the medial frontal cortex in cognitive control. Sci. 2004; 306(5695):443-447.

    [Crossref]  [Google Scholar] [PubMed]

82. Badre D, Nee DE. Frontal cortex and the hierarchical control of behavior. Trends Cogn Sci. 2018; 22(2):170-188.

    [Crossref]  [Google Scholar] [PubMed]

83. Fuster JM. Frontal lobe and cognitive development. J Neurocytol. 2002; 31(3):373-385.

    [Crossref]  [Google Scholar] [PubMed]

84. Hosking JG, Cocker PJ, Winstanley CA. Prefrontal cortical inactivations decrease willingness to expend cognitive effort on a rodent cost/benefit decision-making task. Cereb Cortex. 2016; 26(4):1529-1538.

    [Crossref]  [Google Scholar] [PubMed]

85. Peters J, Büchel C. Overlapping and distinct neural systems code for subjective value during intertemporal and risky decision making. J  Neurosci. 2009; 29(50):15727-15734.

    [Crossref]  [Google Scholar] [PubMed]

86. Weber BJ, Huettel SA. The neural substrates of probabilistic and intertemporal decision making. Brain Res. 2008; 1234:104-115.

    [Crossref] [Google Scholar] [PubMed]

87. Hosking JG, Cocker PJ, Winstanley CA. Prefrontal cortical inactivations decrease willingness to expend cognitive effort on a rodent cost/benefit decision-making task. Cereb Cortex. 2016; 26(4):1529-1538.

    [Crossref] [Google Scholar] [PubMed]

88. Kacelnik A, Bateson M. Risky theories: The effects of variance on foraging decisions. Am Zool. 1996; 36(4):402-434.

    [Crossref] [Google Scholar]

89. Kacelnik A, Bateson M. Risk-sensitivity: crossroads for theories of decision-making. Trends Cogn Sci. 1997; 1(8):304-309.

    [Crossref] [Google Scholar] [PubMed]

90. Caraco T, Martindale S, Whittam TS. An empirical demonstration of risk-sensitive foraging preferences. Anim Behav. 1980; 28(3):820-830.

    [Crossref] [Google Scholar]

91. Marra PP, Francis CM, Mulvihill RS, Moore FR. The influence of climate on the timing and rate of spring bird migration. Oecologia. 2005; 142(2):307-315.

    [Crossref] [Google Scholar] [PubMed]

92. Moore FR, Yong W. Evidence of food-based competition among passerine migrants during stopover. Behav Ecol Sociobiol. 1991; 28(2):85-90.

    [Crossref] [Google Scholar]

93. Simon NW, Montgomery KS, Beas BS, Mitchell MR, LaSarge CL, Mendez IA, et al. Dopaminergic modulation of risky decision-making. J Neurosci. 2011; 31(48):17460-17470.

    [Crossref] [Google Scholar] [PubMed]

94. Ishii H, Ohara S, Tobler PN, Tsutsui KI, Iijima T. Dopaminergic and serotonergic modulation of anterior insular and orbitofrontal cortex function in risky decision making. Neurosci Res. 2015; 92:53-61.

    [Crossref] [Google Scholar] [PubMed]

95. Sugam JA, Day JJ, Wightman RM, Carelli RM. Phasic nucleus accumbens dopamine encodes risk-based decision-making behavior. Biol Psychiatry. 2012; 71(3):199-205.

    [Crossref] [Google Scholar] [PubMed]

96. van den Bos R, Lasthuis W, Den Heijer E, van der Harst J, Spruijt B. Toward a rodent model of the Iowa gambling task. Behav Res Methods. 2006; 38(3):470-478.

    [Crossref] [Google Scholar] [PubMed]

97. Young JW, van Enkhuizen J, Winstanley CA, Geyer MA. Increased risk-taking behavior in dopamine transporter knockdown mice: Further support for a mouse model of mania. J Psychopharmacol. 2011; 25(7):934-943.

    [Crossref] [Google Scholar] [PubMed]

98. St Onge JR, Floresco SB. Dopaminergic modulation of risk-based decision making. Neuropsychopharmacol. 2009; 34(3):681-697.

    [Crossref] [Google Scholar] [PubMed]

99. Simon NW, Gilbert RJ, Mayse JD, Bizon JL, Setlow B. Balancing risk and reward: a rat model of risky decision making. Neuropsychopharmacol. 2009; 34(10):2208-2217.

    [Crossref] [Google Scholar] [PubMed]

100. Simon NW, Montgomery KS, Beas BS, Mitchell MR, LaSarge CL, Mendez IA, et al. Dopaminergic modulation of risky decision-making. J Neurosci. 2011; 31(48):17460-17470.

     [Crossref] [Google Scholar] [PubMed]

101. Fiorillo CD, Tobler PN, Schultz W. Discrete coding of reward probability and uncertainty by dopamine neurons. Sci. 2003; 299(5614):1898-1902.

     [Crossref] [Google Scholar] [PubMed]

102. Jo YS, Heymann G, Zweifel LS. Dopamine neurons reflect the uncertainty in fear generalization. Neuron. 2018; 100(4):916-925.

     [Crossref] [Google Scholar] [PubMed]

103. de Lafuente V, Romo R. Dopamine neurons code subjective sensory experience and uncertainty of perceptual decisions. Proc Natl Acad Sci. 2011; 108(49):19767-19771.

     [Crossref] [Google Scholar] [PubMed]

104. Stefani MR, Moghaddam B. Rule learning and reward contingency are associated with dissociable patterns of dopamine activation in the rat prefrontal cortex, nucleus accumbens, and dorsal striatum. J  Neurosci. 2006; 26(34):8810-8818.

     [Crossref] [Google Scholar] [PubMed]

105. Kahneman D, Tversky A. Prospect theory: An analysis of decision under risk. Econometrica. 1979: 99-127.

     [Crossref] [Google Scholar]

106. Adolphs R, Tranel D, Damasio H, Damasio AR. Fear and the human amygdala. J Neurosci. 1995; 15(9):5879-5891.

     [Crossref] [Google Scholar] [PubMed]

107. Adolphs R. Fear, faces, and the human amygdala. Curr Opin Neurobiol. 2008; 18(2):166-172.

     [Crossref] [Google Scholar] [PubMed]

108. Feinstein JS, Adolphs R, Damasio A, Tranel D. The human amygdala and the induction and experience of fear.                 Curr Biol. 2011; 21(1):34-38.

     [Crossref] [Google Scholar]

109. Ho TH, Lim N, Camerer CF. How “psychological” should economic and marketing models be?.  J Mark Res. 2006; 43(3):341-344.

     [Crossref] [Google Scholar]

110. de Martino B, Camerer CF, Adolphs R. Amygdala damage eliminates monetary loss aversion. Proc Natl Acad Sci. 2010; 107(8):3788-3792.

     [Crossref] [Google Scholar] [PubMed]

111. Charpentier CJ, Martino BD, Sim AL, Sharot T, Roiser JP. Emotion-induced loss aversion and striatal-amygdala coupling in low-anxious individuals. Soc Cogn Affect Neurosci. 2016; 11(4):569-79.

     [Crossref] [Google Scholar] [PubMed]

112. Gelskov SV, Henningsson S, Madsen KH, Siebner HR, Ramsøy TZ. Amygdala signals subjective appetitiveness and aversiveness of mixed gambles. Cortex. 2015; 66:81-90.

     [Crossref] [Google Scholar] [PubMed]

113. Farrar DC, Mian AZ, Budson AE, Moss MB, Killiany RJ. Functional brain networks involved in decision-making under certain and uncertain conditions. Neuroradiol. 2018; 60(1):61-69.

     [Crossref] [Google Scholar] [PubMed]

114. Guo Z, Chen J, Liu S, Li Y, Sun B, Gao Z. Brain areas activated by uncertain reward-based decision-making in healthy volunteers. Neural Regen Res. 2013; 8(35):3344.

     [Crossref] [Google Scholar] [PubMed]

115. Jhang J, Lee H, Kang MS, Lee HS, Park H, Han JH. Anterior cingulate cortex and its input to the basolateral amygdala control innate fear response. Nat Commun. 2018; 9(1):1-6.

     [Crossref] [Google Scholar] [PubMed]