Journal of Depression and Anxiety
Open Access

Structure, Validity and Cut-Off Scores for the APA Emerging Measure of DSM-5 Social Anxiety Disorder Severity Scale (SAD-D)

Kylie Rice^*, Nicola S Schutte, Adam J Rock and Clara V Murray ^*Correspondence: Kylie Rice, School of Psychology, University of New England, Armidale, Australia, Tel: +61-267734259, Email:

Author info »

Introduction

A core feature of social anxiety in the Diagnostic and Statistical Manual Fifth Edition (DSM-5) is the excessive and persistent fear of negative evaluation from others [1]. Cognitions regarding concern over social performance and negative evaluation by others are related to emotional and physiological arousal, and avoidance behaviors, resulting in functional and performance impairments [2]. Social anxiety symptoms can be debilitating, and can lead to a diagnosis of social anxiety disorder [1]. Social anxiety disorder has a lifetime prevalence of around 12%, and is considered to be the third most common mental health disorder [3]. Social anxiety is prevalent across the world, is often diagnosed early in life, and has persistent episodes and a constant course [4]. In addition to the prevalence of this diagnosis, social anxiety symptoms are evident in many presentations and are highly co-morbid with other disorders [4], and elevated social anxiety symptoms are also commonly experienced in non-clinical presentations in the larger population.

Given the importance of social interactions, alongside the prevalence and debilitation of social anxiety symptoms, measures that effectively and efficiently assess the broad range of symptom dimensions are crucial.

Despite the prevalence of symptoms, “there is no standardized approach to the measurement of social anxiety, and researchers and clinicians are faced with numerous scales that purport to measure social anxiety disorder with varying strengths, weaknesses, and psychometric properties” (Sunderland et al., 2018, p. 132) [5]. Most available measures of social anxiety are limited in the range of symptom domains assessed, and most instruments require updating to reflect the DSM-5 social anxiety criteria [6]. In addition, the need for brief social anxiety measures has been identified, with the length of time taken to complete the majority of existing social anxiety measures prohibitive for clinicians and respondents [6]. Time efficient instruments are essential in order to use the measure in both practice and research settings, within a larger battery of tests, and in order to repeat the measure to assess change.

As well as being a diagnostic category, social anxiety can be conceptualized on a continuum of severity [7]. The DSM-5 adopted a more dimensional approach to the classification of psychopathology than previous versions; however, the vast majority of social anxiety measures used in practice and research do not yet reflect this shift [6]. To address this issue, a brief 10-item social anxiety self-report scale (the Social Anxiety Disorder Dimensional Scale; SAD-D) was created to assess severity of social anxiety as a dimension [8]. This scale has been released by the APA as an emerging measure titled the Severity Measure for Social Anxiety Disorder (Social phobia) [9], and has since been named the DSM- 5 Social Anxiety Disorder Severity Scale (SAD-D) [10]. This scale assesses cognitive, physiological and avoidance behavior aspects of social anxiety [8]. The measure was designed for use in clinical practice and research settings to assess symptom severity and change on a dimensional level [10]. The ability to measure symptom change on a dimensional level is clinically useful for treatment and prevention programs, and is an essential feature of the DSM- 5 criteria. In a review of measures, Wong et al. [6] identified the SAD-D as the only social anxiety scale that is based on the DSM- 5 criteria. As such, this scale addresses the limitations of other social anxiety measure [6] as it is dimensional, based on the DSM-5 criteria, and assess a range of symptom domains [8].

LeBeau et al. [10] asked 47 individuals seeking help for social anxiety disorder to complete the SAD-D as well as three other measures of social anxiety and a measure of depression, and to provide information for an Anxiety Disorders Interview Schedule. In this clinical sample of individuals seeking help for social anxiety, the mean score on the SAD-D was 25.7 out of a possible 40. Internal consistency of the SAD-D was 0.86, as assessed by Cronbach’s alpha. Additionally, social anxiety scores were significantly related to three other social anxiety self-report measures and the interviewer ratings on the interview schedule. Lower associations of the SAD-D with the measure of depression than with the measures of social anxiety suggested evidence of discriminant validity. Sunderland et al. [5] administered the SAD-D as well as other social anxiety measures to two large community samples and found high associations of the SAD-D with these other measures.

Thus, the SAD-D was developed as a dimensional assessment of social anxiety and was initially assessed in both non-clinical and clinical samples as part of the development process [8], and then validated with a clinical sample of participants seeking help for social anxiety [10]. Further research suggested that the measure may be used with community samples [5], although examination of the factor structure of the scale with community samples is still needed [11]. Given the measure is intended to comprehensively assess the full range of social anxiety domains [10], the factor structure may be suggestive of the cognitive, physiological and behavioral domains that it is intended to assess [8]. Additionally, further information regarding the internal consistency and validity of the scale using non-clinical samples is needed to assess the usefulness of the SAD-D, and useful criterion cut-off scores have not been established.

The present study

To date, the use of the SAD-D in an Australian sample has not been investigated. As such, a central aim of the present study was to examine the factor structure of the SAD-D using an Australian sample, to assess if the cognitive, behavioral and physiological domains the scale assesses emerged as separate factors and in order to address the identified need to assess the validity of the SAD-D in non-clinical samples [10,11]. Further aims were to examine the internal consistency, along with concurrent, convergent and discriminant validity of the scale in a community sample. Finally, the present study aimed to employ a receiver operating characteristic (ROC) curve analysis to determine the sensitivity (Se) and specificity (Sp) of the SAD-D-10 in predicting fear of negative evaluation, which is a cardinal feature of social anxiety disorder [12,13], and, thus produce a test cut-off value.

Research Methodology

Participants

The university ethics committee approved the research and the study was advertised on an online research board. This recruitment resulted in 997 Australian participants who completed the SAD-D items as well as the validity scales and demographic information. Participants were predominantly mature age (mean age 33.80 yrs; SD=10.28), female (75%), and were studying online (online 97%; on-campus 3%), studying part time (68%), and were geographically located all over the country. While 42% of the participants were looking for alternative full time (19%) or part time work (23%), 92% of participants were currently engaged in paid employment as well. Half of all participants were already working part time (50%), and 42% were currently engaged in full time employment. The majority of participants (75%) had dependent children (none 25%; one child, 21%; two children, 33%; three children 17%; four or more 4%). Thus, these sample characteristics indicated the majority of the participants were mature-age, part-time students who were also working in paid employment, representing a nonclinical, community sample. Participants completed the measures via an on-line survey, following the provision of implied consent.

Measures

DSM-5 Social Anxiety Disorder Severity Scale (SAD-D)

This 10-item scale assesses severity of social anxiety on a continuum [8], including the cognitive, physiological and avoidance behavior symptoms of social anxiety [10]. Respondents rate how frequently they have experienced each symptom in social situations over the past week on a 5-point scale on which a rating of 0 indicates ‘Never’ and a rating of 4 indicates ‘All of the time’. Scores range from 0 to 40, with higher scores indicating greater severity. The 10 items are shown in Table 1. In the present study, the mean score was 10.8 (SD=8.89) and internal consistency assessed by Cronbach’s alpha was 0.95.

Table 1: Factor loadings on the Ten-Item Social Anxiety Scale (SAD-D).

Depression, Anxiety, and Stress Scales (DASS-21)

The Depression, Anxiety, and Stress Scales (DASS-21) [14] assess symptoms of anxiety as well as symptoms of depression and stress that respondents have experienced in the past week. Previous research indicates that the scale and subscales have good reliability and evidence of validity [14]. Higher scores indicate a higher level of symptoms. In the present study, means and standard deviations were as follows: anxiety (M=3.33, SD=4.27); depression (M=6.27, SD=4.67); and stress (M=4.04, SD=4.67). Internal consistency for the present study was assessed by Cronbach’s alpha, as 0.89 for anxiety, 0.93 for depression, and 0.89 for stress. As well as comprising separate scales, a composite of the 21 items can be used as an indicator of general mental health [15].

Fear of Negative Evaluation

The Brief Fear of Negative Evaluation (BFNE-S) scale consists of eight items assessing typical worry regarding being negatively evaluated by others, which is symptomatic of social anxiety [16]. Prior research indicates the scale demonstrates good internal consistency and validity [16], and higher scores indicate greater fear of negative evaluation. In the present study the mean score was 28.50 (SD=11.72). Internal consistency assessed by Cronbach’s alpha was 0.97.

Results

Exploratory and confirmatory factor analysis of the DSM- 5 Social Anxiety Disorder Severity Scale (SAD-D) Items

An approximately random split of the data set into halves allowed exploratory and confirmatory factor analyses to be conducted on separate subsamples of 495 and 514 participants who completed all SAD-D items. These sample sizes were adequate for factor analysis, meeting Tabachnick & Fidell's (2019) recommendation of a minimum of 300 cases [17] . For the exploratory factor analysis (EFA), Maximum Likelihood factor analysis examined the factor structure of the items. Bartlett’s test of sphericity was significant, p<0.0001, and the Kaiser-Meyer-Olkin (KMO) index was 0.94, exceeding the recommended minimum of 0.6, indicating the data was suitable for factor analysis. The scree plot suggested a one-factor solution, with only the first factor having an eigenvalue over one. The first factor, with an eigenvalue of 6.87 explained 69 percent of the variance. The second and third factors had eigenvalues of 0.69 and 0.67, respectively. Such a strong loading on only one factor indicates that a rotation is not suitable, thus the extracted maximum loadings on the first factor were deemed to be the best representation of the structure of the items. Table 1 shows the items and their loadings on this factor.

The high factor loadings of all items on one factor suggested that it might be possible to reduce the 10 items comprising the SAD-D to create an even briefer scale with good psychometric properties. The six items (items 2, 3, 4, 5, 6, and 9) with the highest factor loadings were selected for this abbreviated scale. The items reflected affective, cognitive and physiological elements of social anxiety and all had loadings of 0.65 or higher. This six-item scale had a reliability, as assessed by a Cronbach’s alpha, of 0.93. We have termed this scale the Brief DSM-5 Social Anxiety Disorder Severity Scale (SAD-D-6). The mean score on this scale was 6.79 (SD=5.51).

Confirmatory factor analyses (CFA) of the 10-item scale (SAD-D) showed the following model fit indices: CFI=0.92, TLI=0.88, and RMSEA=0.13 (95% CI=0.12, 0.15). Thus, based on the responses of this sample of participants, the CFA fit indices suggested a marginally acceptable fit with the structure suggested by the EFA of responses of the first sample of participants. CFA of the six-item scale (SAD-D-6) showed slightly better indices than for the 10-item scale as follows: CFI=0.96, TLI=0.93, and RMSEA=0.12 (95% CI=0.10, 0.14).

Validity of the SAD-D-10 and the SAD-D-6

Scores on both the original SAD-D-10 and the brief SAD-D-6 scale showed strong significant positive associations with general anxiety and fear of negative evaluation (Table 2). The associations of both the 10-item scale and the six-item scale with the concurrent validity measures were almost identical, suggesting equivalent validity of the two versions as based on these associations.

Table 2: Associations of the SAD-D-10 and the SAD-D-6 with general anxiety, fear of negative evaluation, depression and stress.

Associations of the SAD-D-10 and the SAD-D-6 with depression and stress were also significant and high. As expected, the correlation between the SAD-D-10 and general anxiety was significantly larger than the correlation between the SAD-D-10 and depression (z=2.09, p=0.02), providing some evidence of discriminant validity. The correlation between the SAD-D-130 and anxiety was significantly larger than the correlation between the SAD-D-10 and stress (z=1.76, p=0.04). Similarly, the association between the SAD-D-6 and general anxiety was significantly larger than its respective associations with depression (z=2.06, p=0.02) and stress (z=1.72, p=0.04), again providing some evidence of discriminant validity.

Identifying high levels of distress on the SAD-D-10

LeBeau et al. [10] reported that individuals seeking help for social anxiety had a mean SAD-D-10 score of 25.7. This mean score obtained in a clinical sample might indicate a distressing level of social anxiety among Australian university participants, and if so, participants scoring at this level might be expected to be experiencing poor mental health. To test this proposal, the present sample was divided into those scoring 25.7 and above on severity of social anxiety on the SAD-D-10, or below. In a comparison of composite DASS-21 scores, an indicator of general mental health, the 68 individuals scoring at 25.7 and above on the SAD-D-10 had significantly poorer mental health (M=33.03, SD=16.89) than those scoring below (M=11.80, SD=10.24), t(951)=15.56, p=0.0001, mean difference=21.36 (95% CI=23.9, 18.55), t(951)=15.56, p=0.0001, 2000 bootstraps as per Pek et al. [18].

ROC curve analysis

The SAD-D-10 demonstrated strong predictive ability regarding fear of negative evaluation with 88% Se and 85% Sp, which yielded a test cut-off value of 18.50, rounded up to 19. In Figure 1, the red circle denotes this cut-off point which produced the maximum Youden’s index (Se + Sp – 1) of 0.72, corresponding “to a point on the ROC curve with the highest vertical distance from the 45° diagonal line” (Habibzadeh et al., 2016, p. 299). The area under the curve (AUC) for this logistic regression model was 0.92 (p<0.001 versus the null hypothesis where the true area=0.5; 95% CI=0.88, 0.95), thus, indicating significant predictive ability (Figure 1).

Figure 1: ROC curve of the SAD-D-10’s predictive value for fear of negative evaluation.

The SAD-D-6 also demonstrated strong predictive ability regarding fear of negative evaluation with 90% Se and 81% Sp, which produced a test cut-off value of 10.50, rounded up to 11. The red circle in Figure 2 denotes this cut-off point which produced the maximum Youden’s index of 0.71. The AUC for this model was 0.92 (p<0.001; 95% CI=0.88, 0.95), thus, indicating significant predictive ability (Figure 2).

Figure 2: ROC curve of the SAD-D-6’s predictive value for fear of negative evaluation.

Discussion

The present study examined the utility of the APA’s emerging dimensional measure of social anxiety, the Severity Measure for Social Anxiety Disorder (Social Phobia) (SAD-D) [9]. The result of exploratory and confirmatory factor analyses of responses from a large Australian non-clinical sample suggested that the measure is unidimensional, indicating that the scale assesses severity of social anxiety as a uniform construct. The post-hoc results of the factor analyses suggested that the brief 10-item SAD-D scale can be shortened to a six-item scale while still retaining good internal reliability. The internal consistency of the original SAD-10 scale was 0.95 and the internal consistency of the new SAD-D-6 scale was 0.93. Thus, both versions of the scale have excellent internal reliability. Both the original SAD-10 scale and the new SAD-D-6 scale showed evidence of concurrent validity through associations with symptoms of general anxiety and fear of negative evaluation. Both scales showed some evidence of divergent validity through weaker associations with depression and stress than with general anxiety. These findings are consistent with concurrent validity evidence for the SAD-10 scale provided by LeBeau et al. [10] and Sunderland et al. [5]. A comparison of individuals who scored at or above versus below the mean SAD-D-10 score reported by LeBeau et al. [10] with their clinical sample, showed that those who scored at or above this mean had significantly poorer general mental health than those who scored below, providing evidence of convergent validity for the SAD-D-10. This research represents the first investigation of the use of this measure in an Australian sample.

The ROC curve analyses showed that both the SAD-D-10 and SAD-D-6 demonstrated significant predictive ability regarding the fear of negative evaluation, and, thus, may be considered appropriate measures to dichotomize a non-clinical Australian sample into those who fear negative evaluation and those who do not fear negative evaluation, which is a central and distinctive feature of social anxiety disorder [12,13]. The scales’ ability to predict fear of negative evaluation suggests that the scales have utility in detecting social anxiety symptoms and differentiating these from general anxiety and/or psychological distress. The point on each of the two ROC curves where the Youden’s index was maximum was used to generate test cut-off values for the SAD-D-10 and SAD-D-6, which were 19 and 11, respectively. For the SAD-D-10 the Se was 88% and the Sp was 85%. Consequently, the model produced marginally fewer false negative results than false positive results. Similarly, for the SAD-D-6 the Se was 90% and the Sp was 81%. Therefore, this model also generated fewer false negative results than false positive results. Importantly, the SAD-D-6 constitutes a more parsimonious measure relative to the SAD-D-10 and also provides an increase in Se compared to the SAD-D-10. The development of more parsimonious measures of social anxiety that are accompanied by an increase in Se has been advocated by researchers [19]. Similar analyses proved useful in developing the well-used shortened versions of the commonly used Fear of Negative Evaluation scale that was originally presented in a 30-item version, then a 12-item brief version, and an even briefer eight-item version [16].

The results of this study provide support for the SAD-D-10 and the SAD-D-6 as measures of severity of social anxiety that overcome some limitations of existing measures. As noted by Wong et al. [6], the majority of existing social anxiety instruments are not aligned with current DSM-5 criteria, are prohibitively long for respondents and clinicians, and with the exception of the SAD-D, do not take a dimensional approach. Both the SAD-D-10, and the shorter SAD-D-6, provides viable and efficient social anxiety measures that are dimensional and are based on the DSM-5 criteria.

Limitations of the research include the majority of participants were female, and that the sample was drawn from one university population in one country of interest. Concerns have been raised that university student samples are not necessarily generalizable to other populations [20]. However, while all participants in this study were engaged in some form of study, 92% of participants were also working in paid employment, 75% of participants had one or more dependent children, and the average age of 33.8 years was indicative of a mature age sample. Thus, the sample characteristics more closely represented a non-clinical, community sample, rather than the traditional early-adult on campus undergraduate students. Furthermore, other researchers have argued that student samples do not inherently present an issue for external validity [21], and research involving both university students and other adults failed to find wide-spread differences in the cohorts [22]. As such, the fact that participants were enrolled in a university course does not necessarily limit the findings, although it would be interesting for future research to evaluate the use of the SAD-D-10 and the SAD-D-6 scales in a dual sample of both university students and non-university students to assess any possible sample differences.

Future research might also investigate the cross-cultural relevance of both the SAD-D-10 and the SAD-D-6 scales. The applicability of the test cut-off values identified in the present study for predicting fear of negative evaluation could be usefully re-assessed within a clinical sample. Further, the validity analyses of SAD-D-10 and the SAD-D-6 scales were based on concurrent data collection. Same method variance may therefore have influenced the results. Longitudinal research is needed to assess changes over time and the effects of repeated measurement, specifically incorporating other methods of measuring symptom severity, such as clinical interviews. Future research might also investigate the scales’ sensitivity to change in treatment and prevention programs.

The present study provides support for the use of the SAD-D-10 and the new SAD-D-6 in both research and practice. The establishment of test cut-off values in the present study may also be useful for future research purposes in which it is beneficial to assess respondent scores in relation to a criterion cut-off score. The test cut-off values also may be useful for practice settings, providing a possible screening tool for the dimensional assessment of social anxiety symptoms. The dimensional nature of these scales will allow clinicians to compare client scores from one time to another, enabling clinicians to assess the fluctuation in social anxiety symptoms in response to treatment, in progress monitoring and assessment of maintenance of gains.

Conclusion

In conclusion, both the original SAD-D-10 scale and the new SAD-D-6 scale assess the affective, cognitive and behavioral components of severity of social anxiety as a unidimensional construct. Both versions of the scale showed evidence of reliability and validity. However, the SAD-D-6 is a more parsimonious measure that provides an increase in sensitivity compared to the SAD-D-10. These brief scales may have utility in treatment and prevention efforts focused on ameliorating social anxiety, as well as in research.

References

1. American Psychiatric Association. Diagnostic and statistical manual of mental disorders (5th ed.). American Psychiatric Association. 2013.
2. Heimberg RG, Brozovich FA, Rapee RM. A cognitive-behavioral model of social anxiety disorder. InSocial Anxiety. Academic Press. 2014;705-728.
3. Kessler RC, Berglund P, Demler O, Jin R, Merikangas KR, Walters EE. Lifetime prevalence and age-of-onset distributions of DSM-IV disorders in the National Comorbidity Survey Replication. Arch Gen Psychiatry. 2005;62(6):593-602.
4. Stein MB, Fuetsch M, Müller N, Höfler M, Lieb R, Wittchen HU. Social anxiety disorder and the risk of depression: a prospective community study of adolescents and young adults. Arch Gen Psychiatry. 2001;58(3):251-256.
5. Sunderland M, Batterham P, Calear A, Carragher N, Baillie A, Slade T. High agreement was obtained across scores from multiple equated scales for social anxiety disorder using item response theory. J Clin Epidemiol. 2018;99:132-143.
6. Wong QJ, Gregory B, McLellan LF. A review of scales to measure social anxiety disorder in clinical and epidemiological studies. Curr Psychiatry Rep. 2016;18(4):38.
7. LeBeau R, Bögels S, Möller E, Craske M. Integrating dimensional assessment and categorical diagnosis in DSM-5: The benefits and challenges of the paradigm shift for the anxiety disorders. Psychopathol Rev. 2015;2(1):83-99.
8. Lebeau RT, Glenn DE, Hanover LN, Beesdo-Baum K, Wittchen HU, Craske MG. A dimensional approach to measuring anxiety for DSM-5. Int J Methods Psychiatr Res. 2012;21(4):258-72.
9. Craske M, Wittchen U, Bogels S, Stein M, Andrews G, Lebeu R. Severity measure for social anxiety disorder. American Psychiatric Association. 2013.
10. LeBeau RT, Mesri B, Craske MG. The DSM-5 social anxiety disorder severity scale: Evidence of validity and reliability in a clinical sample. Psychiatry Res. 2016;244:94-96
11. Haelle, T. Social Anxiety Disorder Self-Reported Severity Scale validated in English-speaking adults. Psychiatry Advisor. 2016;47(11):1915-1921.
12. Clark DM, Wells A. A cognitive model of social phobia. Social phobia: Diagnosis, assessment, and treatment. The Guilford Press, USA. 1995;69-93.
13. Rapee RM, Heimberg RG. A cognitive-behavioral model of anxiety in social phobia. Behav Res Ther. 1997;35(8):741-756.
14. Henry JD, Crawford JR. The 21-item version of the Depression Anxiety Stress Scales (DASS–21): Normative data and psychometric evaluation in a large non-clinical sample. Br J Soc Clin Psychol. 2005;44(22):227-239.
15. Zanon C, Brenner RE, Baptista MN, Vogel DL, Rubin M, Al-Darmaki FR, et al. Examining the dimensionality, reliability, and invariance of the Depression, Anxiety, and Stress Scale–21 (DASS-21) across eight countries. Assessment. 2020.
16. Carleton RN, Collimore KC, McCabe RE, Antony MM. Addressing revisions to the Brief Fear of Negative Evaluation scale: Measuring fear of negative evaluation across anxiety and mood disorders. J Anxiety Disord. 2011;25(6):822-828.
17. Tabachnick BG, Fidell LS. Using multivariate statistics (7th edn). Allyn & Bacon, USA. 2019.
18. Pek J, Wong O, Wong A. How to address non-normality: A taxonomy of approaches, reviewed, and illustrated. Front Physiol. 2018;9:2104.
19. Carleton RN, Collimore KC, Asmundson GJ. Social anxiety and fear of negative evaluation: Construct validity of the BFNE-II. J Anxiety Disord. 2007;21(1):131-141.
20. Kam C, Franzese, RJ. Modeling and interpreting interactive hypotheses in regression analysis. University of Michigan Press. 2007.
21. Druckman JN, Kam CD. Students as experimental participants. Cambridge Handbook of Experimental Political Science. 2011;1:41-57.
22. Snook AK. Defense of external validity: Do college student samples yield different results? Doctoral Dissertations. University of Connecticut, USA. 2011.