Journal of Psychology & Psychotherapy
Open Access

Psychometric Properties of the Experiences in Close Relationship Scale (Short Form) with Pakistani Adolescents

Somia Imran^*, Angus MacBeth, Ethel Quayle and Stella W.Y. Chan ^*Correspondence: Somia Imran, Department of Clinical Psychology, School of Health in Social Science, University of Edinburgh, Doorway 6, Medical Quad, Teviot Place, Edinburgh EH8 9AG, UK, Tel: 07724044685, Email:

Author info »

Introduction

The high prevalence and burden of mental health difficulties in young people and adolescents present a global public health priority [1]. The majority of studies related to youth mental health have been conducted in high-income countries and there is a paucity of research conducted in middle and low-resource settings, such as South Asia [1,2]. There is even less research on psychological processes underpinning youth mental health risk and resilience in such settings, raising issues around the cultural appropriateness or invariance of measurements across cultures. Notably, although 38% of Pakistan’s population is under 18 years, research exploring mental health in this age group is scant, partially as adolescence was only recently accepted as a distinct developmental stage in Pakistani culture [3]. One obstacle to developing research in this context is a lack of psychometric measures for use with adolescents [4], thus it is important to develop and test psychometrically robust, culturally adapted measures for use with Pakistani youth.

Prior research indicated two methodological approaches, namely emic versus etic, with regard to how psychological concepts can be measured across cultures [5]. An emic approach implies the development of indigenous measures in line with language and cultural characteristics, as well as the employment of interviews and/or observation to explore hidden nuances [6]. However, this strategy does not allow for comparison across cultures of the variables being measured [7]. Ethic approaches assume the universality of human behaviors and thereby the same measurements could be used across cultures as a means of identifying common characteristics [6]. Nevertheless, this approach may undermine cultural differences and the relevance and validity of the measure/s being used are at risk. Further, by employing etic methods it cannot be confirmed whether the differences are due to the measurement variance or due to actual differences in associations among psychological variables across cultures [8,9].

The current research took a mid-way stance using a derived etic approach involving the adaptation of psychometrically sound measures rather than developing indigenous scales [8-10]. As such, this was thought to be one of the most efficient and cost effective methodological developments [4]. The goals of adaptation of instrument techniques, such as back translation methods, are similar to other techniques in psychological research ensuring reliability, validity and increased generalisability of the findings [5]. However, these goals may be hard to achieve since, alongside cultural differences, there are language differences as well [11]. Of importance, there is no consensus regarding the step-by-step guidelines to ensure semantic equivalence of the items and sound psychometric properties such as what type of validity of the translated version has to be established [12]. According to the available evidence, to ensure a balanced treatment of linguistic, cultural, contextual and scientific information, usually, a team of experts are recommended to overcome theoretical, linguistic and practical biases [13].

Furthermore, prior research has suggested some further steps to promote cultural sensitivity, such as evaluation of the self-reports by the target population as part of the translation process [12] and evaluation of emic versus etic aspects of the items [5]. In the current study, for both of these purposes, two bilingual adolescents and one anthropologist were included in the translation committee to ensure the clarity and cultural relevance of the items respectively [12]; see details in methodology). Additionally, short forms of the assessment tools with sound psychometric properties were used as the preferred option, in order to carry out an evaluation of different variables within a reasonably short testing time simultaneously [2].

The current study specifically focuses on the Urdu-language validation of attachment – a key psychological construct for adolescent development and mental health. In attachment theory, individuals internalize attachment encounters in terms of availability and responsiveness with primary attachment figure/s, with high availability and responsiveness precipitating attachment security [14]. In adolescence, secure attachment is associated with greater resilience and wellbeing [2]. A number of self-report instruments for adolescent attachment have been validated for use with Pakistani adolescents such as Inventory of Parents and Peer Attachment (IPPA) [15] and Parental Bonding (PBI) [16]. However, these measures either only assess attachment within restricted relationships (IPPA), or are in fact measures of recall of parenting, rather than attachment (PBI). As the attachment system expands to multiple relationships in adolescence [17], there is a need for culturally adapted, psychometrically robust, general attachment measures for Pakistani adolescents.

One option is the Experiences in Close Relationship Scale (ECR) [18] - a two factors measure of attachment-anxiety (fear of abandonment, an excessive need for approval, and distress of unavailability of others within relationships) and attachmentavoidance (fear of dependence, an excessive need for selfreliance, and reluctance to self-disclose within relationships). Short versions of the original 36-item version have been developed and validated to use for youth such as the 12-items Revised-General-Short-Form of ECR-12 [19].

Given the importance of attachment for adolescent resilience, and the ubiquity of the ECR the current study aimed to translate and validate this measure for use with Pakistani adolescents.

Research Methodology

Ethics approval

Ethical approval for the current study was obtained from the research ethics committee of the University of Edinburgh, Fatima Jinnah Women University and the local education authority in Rawalpindi, Pakistan.

Participants

A total of 400 adolescents (12-18 years; mean age 14.5, SD 1.64; 51.5% male) were recruited from three (randomly selected) schools. A priori power calculation indicated that a minimum sample size of 300 based on a ratio of 15 cases per item [20] is required; hence the current sample size (400) is sufficiently powered. In terms of religion, the majority 93.8% were Muslims and 6.0% were Christians. For ethnicity, 79.8% were Punjabi, 4.0% Sindhi, 1.3% Balochi, 9.8% Pathans and 5.3% were from other ethnic groups. Family Affluence Scale-II (FAS) [21] was used to measure socio-economic status (SES). The majority of participants (64.5%) were from middle SES, with 21.0% from low SES, and 14.5% from high SES families.

Measures

The demographic questionnaire included questions about gender, age, grade, ethnicity, relationship status, religion and family-system. Family Affluence Scale (FAS II) [21,22] was also included. It has four items:

• Does your family own a car, van or truck?

• Do you have your own bedroom for yourself?

• During the past 12 months, how many times did you travel away on holiday with your family?

• How many computers does your family own?

These items assess adolescent socio-economic status based on objective and subjective measures of family wealth [22]. Based on FAS II total score, a score of 0‐3 indicates low affluence, a score of 4‐6 indicates middle affluence, and a score of 7‐9 represents high affluence.

The Experiences in Close Relationship Revised-General-Short- Form (ECR-R-GSF-12) [19]. It was used to measure adolescents` attachment with their primary caregiver with the instructions: “Think about a person/s who has been involved in providing you basic needs and you think about and go to that person/s when you are distressed/anxious for example mother, father or carer”. This 12-item scale has two subscales: attachment-anxiety and attachment-avoidance. Participants rate each question on a 7-point Likert scale, ranging from 1= “ disagree-strongly ” to 7= “ agree-strongly ” . Higher scores indicate higher anxiety or avoidance. Overall, there are four positively worded (item example: I do not often worry about being abandoned) and eight negative worded items (item example: I try to avoid getting too close to my primary caregiver/s). Wei et al. [19] found coefficient alphas 0.78 for anxiety and 0.84 for avoidance in their study.

The Relationship Questionnaire (RQ) [23]: This measure consists of four paragraphs indicating a specific attachment pattern: secure, preoccupied, dismissing and fearful. These attachment patterns correspond to a two-dimensional attachment model underlying the ECR-12, such as secure (low anxiety, low avoidance), preoccupied (high anxiety, low avoidance), fearful (high anxiety, high avoidance) and dismissing (low anxiety, high avoidance). Participants use a 7-point Likert scale, ranging from 1 (strongly disagree) to 7 (strongly agree) to rate each paragraph. In the current study, the RQ was adapted to measure attachment with the primary caregiver using the same instructions as the ECR-12. An Urdu RQ has been used with Pakistani older adolescents by Pervaiz and Sohail [24] and Adil and Kamal [25] respectively. Schmitt and colleagues [26] validated RQ in 62 cultures but due to forced choices for attachment patterns, it is considered relatively less reliable (α 0.32-.79) and is mostly used for secondary investigation (i.e., validation) or in combination with multi-item measures [14].

The Hospital Anxiety and Depression Scale (HADS) [27]: It was used to measure depression and anxiety symptoms. It is a 14- item scale with two subscales: depression and anxiety (7-item each). `Participants were asked to rate each item based on how they feel generally on a 4-point Likert scale. HADS has good psychometric properties [28]. This study used the extensively validated Urdu HADS [29].

The Rosenberg`s Self-Esteem Scale (RSES) [30]: It was used to measure self-esteem in adolescents. It has 10 items and participants rate each statement on a 4-point Likert scale from strongly-agree to strongly-disagree. Higher scores indicate high self-esteem. The Urdu RSES has been validated with Pakistani adolescents (α=0.77) [31].

Procedure

The study was conducted in two phases: Translation-phase and Validation-phase

Translation-phase

Permission to translate ECR-12 was obtained from the corresponding author. A committee of seven members (the researcher, psychologist, anthropologist, bilingual, two adolescents and a linguist) was arranged for evaluation of the Urdu versions. Following Brislin`s [5] guidelines, to ensure measurement invariance [32], the committee members judged items in terms of

Semantic equivalence: The same meaning and no grammatical errors.

Experiential equivalence: Whether the item is applicable in the new culture and

Conceptual meaning: Which seeks to ensure that each item assesses the same aspect in different cultures.

Following the six steps suggested by Borsa et al. [12], a modified version of a translation procedure was followed. The translation procedure was completed in four months.

Forward translation stage: Firstly, two forward translations were independently completed by two translators (the researcher and one committee member). One translator (researcher) translated scales while considering how the construct (such as attachment) of the scale was measured. The second translator (one committee member unaware of the assessed construct), translated the scale while considering the language component of the scale.

Synthesis stage: Both forward translations were compared to identify discrepancies and after an agreement was achieved between the two translators, the synthesized version was developed.

Evaluation stage one: At this step, the synthesized version was evaluated first by the committee members then by the target population: a focus group was conducted with a group of 5 adolescents (not part of the committee) to ensure the text was accurate, readable and understandable as well as to identify ambiguous instructions. Whenever adolescents showed difficulty or pointed out a lack of clarity, they were encouraged to provide the reason, and then items were discussed with the committee members. After discussion, it was decided that it would be pragmatic to incorporate English and Urdu alternate words within brackets for the words and phrases that were identified as difficult during the focus group. Further, a conceptual misunderstanding was seen in relation to one item in ECR-12 (Anxiety subscale; item 5), “I get frustrated if primary caregiver/s is not available when I need them”. The focus group members expressed that they had assumed that the frustration was because of missing their primary caregiver, so they scored high on this item, but not because they felt some kind of insecurity. The anthropologist (committee member) suggested that this item may have some etic aspect (not expected in western society but expected in eastern collectivist society) thereby the target population was understanding it as an expression of “healthy attachment” not “ insecurity” . Recognizing this, a phrase “due to lack of trust” was added within brackets (such as “I get frustrated (due to lack of trust) if primary caregiver/s is not available when I need them”) to ensure the cultural adaptability as well as conceptual accuracy.

Back translation stage: The synthesized version was then given to two back-translators (not part of the committee) who were blind to the original survey with the aim of evaluating the extent to which the translated version reflected the item content of the original version.

Evaluation stage two: Two back translated versions alongside the synthesized version were then forwarded to the committee for stage two of the evaluation. No specific problems were highlighted at this stage. After making the required adjustments to the font size, layout and structure as per the adolescent committee members` suggestions, the entire booklet of the questionnaire was pilot tested.

Pilot Testing: This final stage of the translation phase was completed in two phases

• All questionnaires were administered in an interview format to 5 adolescents and

• A small pilot study (try-out) was sampled with 20 school adolescents in a group setting to estimate time duration to complete the questionnaires and to troubleshoot any problems. Urdu versions were finalized based on the agreement of committee members and modifications were made following the pilot testing.

Validation-phase

After receiving permission from school principals, an information sheet, and parental opt-out consent form was sent to parents of all adolescent students in three participating schools. In addition to parental consent, adolescent participants were also asked to complete a consent form prior to participation. Completion of the questionnaires took approximately 25 - 30 minutes, after which participants were debriefed orally.

Data Analysis

Descriptive, correlation and reliability analyses were conducted in the Statistical Packages for Social Sciences version 21 (SPSS). Confirmatory Factor Analysis (CFA) was tested using Mplus version 6.2 [33]. Prior to CFA analysis, data were screened for multivariate normality and missing values. There were no missing values as incomplete questionnaires (n=30) were excluded before the data entry. Reliability was assessed using Cronbach’s alphas (0.7-0.8 acceptable, 0.8-0.9 good and > 0.90 excellent) [34]. Correlation coefficients were used to determine the validity of the Urdu versions. It was hypothesized that the ECR-12 anxiety and avoidance subscales will correlate with the RQ ratings, indicating both convergent and discriminant validity. ECR-12 scores` (total and subscales) positive correlation with HADS scores (total and subscales) and a negative correlation with RSES total score will indicate good construct validity.

For CFA, as suggested by Hu and Bentler [35], four indexes were used to assess the goodness of fit of the models: the comparative fit index (CFI; values of 0.90 or greater), Tucker-Lewis index (TLI; values of 0.90 or greater), the root mean square error of approximation (RMSEA; values of 0.06 or less), and the standardized root mean square residual (SRMR; values of 0.08 or less) indicate a good fitting model.

As part of ECR-12 CFA models, the possible influence of response sets was also evaluated. Response sets arise when a scale has both negatively and positively worded items that may lead participants to respond according to the direction of item wording rather than its content. This potentially confounding factor was avoided by adding two orthogonal method factors, where all positively worded items are loaded on one factor and all negatively worded items are loaded on another factor (Figure 1) [19].

Figure 1: Two oblique factors (12 items) with two orthogonal positively and negatively worded factors. #=item number; r=items with reversed score (Wei et al., 2007).

Results

Descriptive statistics of measures are presented in Table 1.

Table 1: Descriptive Statistics of the Measures (N=400). Note: N=Number of participants, α=Cronbach’s alpha, M=Mean, SD=Standard deviation, ECR-12=Experiences in Close Relationships Scale, HADS=Hospital Anxiety and Depression Scale, RSES=Rosenberg’s Self-Esteem Scale.

High Cronbach alphas (.81-.91) were found across all measures. Skewness and kurtosis were within normal ranges [36]. For construct validity, results were consistent with hypotheses. The ECR-12 anxiety and avoidance subscales showed positive significant correlations with RQ preoccupied, fearful and dismissing subscales and negative significant associations with RQ secure subscale (Table 2). ECR-12 total and subscales` scores were significantly positively correlated with the HADS total and subscales and negatively correlated with RSES total score (Table 2).

Table 2: Correlations of the ECR-12 total and subscales, with RQ subscales, HADS total, subscales, and RSES total (N=400). Note: ECR=Experiences in Close Relationship Scale-12, ECR-An=ECR-Anxiety subscale, ECR-Av=ECR-Avoidance subscale, RQ=Relationship Questionnaire. RSES=Rosenberg’s Self-Esteem Scale. Bold indicates *=p < 0.05; **=at p < 0.004 after Bonferroni correction was applied (.05 /11).

Confirmatory Factor Analysis (CFA)

Three CFA models were tested following Wei et al. (2007). Model 1 involved two oblique factors (anxiety and avoidance) with 6 items loading on the anxiety factor and 6 items loading on the avoidance factor. In model 2, along with two oblique factors, one method factor (positively-worded) was added. In Model 3, beside two oblique factors, two orthogonal method factors (both positively and negatively worded; (Figure 1) were added. In the current sample, Model 3 revealed the best fit to the data (Table 3). All factor loadings were significant (Table 4). Residual terms were not correlated and the latent variable variances were fixed to one in all models.

Table 3: Confirmatory Factor Analysis of ECR-12 (N=400). Note: RMSEA=root mean square error of approximation, CFI=comparative fit index, TLI=Tucker-Lewis index, SRMR=standardized root mean square residual, df=degree of freedom, CI= confidence interval X²=Chi-square. Bold denotes *=p < 0.05.

Table 4: Final model factor loadings of ECR-12 anxiety and avoidance subscales (N=400). **=p < 0.01.

Discussion

The current study examined the psychometric properties of an Urdu translation of the ECR-12 in Pakistani adolescents. Results indicate that the Urdu translation of this measure has psychometric properties comparable to the original ECR-12.

There was a positive skew in responses to the ECR-12 items indicating that very few adolescents reported high anxiety or high avoidance related attachment insecurity. Future investigation with clinical or purposive sampling with the potentially insecure sample will provide further understanding into this [14]. Overall, the avoidance subscale items showed high factor loadings (Table 4) than the anxiety subscale items. Of interest, anxiety subscale item 5 showed the least factor loading (.33) which could be due to the conceptual misunderstanding regarding healthy attachment as pointed out by the adolescent translation committee members. Urdu ECR-12 total scale and subscales showed high values of Cronbach alphas comparable to the internal consistency of the original ECR-12 [19]. This could be due to the considerations are taken during the translation process, for instance, to ensure content and language appropriate for the target population adolescents were included in the translation committee alongside experts (see method for details). For rigorous pilot testing, both qualitative (focus group and interview format) and quantitative strategies (pilot testing in a small group) were employed. As such, these results suggested that the Urdu version retained the internal consistency.

Regarding structural validity, ECR-12 CFA results are consistent with a model including two latent factors plus two methodfactors, providing comparably good fit to the data for both the short (12-item) and original (36-item) versions of the ECR [19]. Methods-factors were included to overcome the influence of systematic errors on responses for positively or negatively worded items. Addition of these method factors preserves the factor structure and number of items included in each subscale. It has been suggested that responses are more reliable when positive items are presented with other positive items and negative items with other negative items rather than in a mixed format [37]. Future studies could investigate how to eliminate item wording effects via improved data collection procedures. Overall, these results support that anxiety and avoidance represent fundamental features of the quality of attachment across cultures.

For construct validity, the avoidance and anxiety subscales of the ECR-12 were significantly correlated with RQ ratings, indicating convergent and discriminant association [14]. Interestingly, all RQ ratings showed the same strength of correlation with both anxiety and avoidance, perhaps due to the forced choices format of the RQ or limited interpretation of each paragraph. Results are also consistent with the observation that attachment styles in the RQ vary across cultures [26].

ECR-12 total score and subscales also showed positive significant associations with HADS total and subscales. These results support the well-established position that attachment insecurity associates with higher mental health symptoms [19]. Conversely, ECR-12 scores were significantly negatively associated with RSES scores, consistent with existing evidence for the association between secure attachment and high self-esteem in adolescents [14].

Conclusion and Limitations

There are several limitations to acknowledge. Data were collected using self-report measures which may be prone to bias. Future research could employ multiple informant approaches for data collection. The sample mainly comprised adolescents with middle SES and from mainstream schools; future studies with wider representation of family-affluence and clinical populations would help increase generalizability. Test-retest reliability is also required to examine longitudinal stability of ECR-12. Nevertheless, the current study provides evidence for sound psychometric properties of the Urdu ECR-12 and facilitating its use in future research. Its robust psychometrics permits crosscultural comparison and further use in Urdu-speaking samples.

Acknowledgements

Authors thank all the study participants, translation committee members and the ethics committee of Fatima Jinnah Women University Pakistan. This work was supported by the College Research Awards, University of Edinburgh, UK.

References

1. Camilletti E. Realizing an enabling environment for adolescent well-being: an inventory of laws and policies for adolescents in South Asia, UNICEF Office of Research – Innocenti, Florence, Italy, 2018.
2. Imran S, Macbeth A, Quayle E, Chan S. Secondary attachment and mental health in Pakistani and Scottish adolescents: A moderated mediation model. Psychology and Psychotherapy: Theory, Research and Practice, 2020:e12280.
3. Qidwai W, Ishaque S, Shah S, Rahim M. Adolescent lifestyle and behaviour: A survey from a developing country. PLOS One. 2010; 5(9):e12914.
4. Khalid A, Qadir F, Chan S, Schwannauer M. Parental bonding and adolescents’ depressive and anxious symptoms in Pakistan. J Affective Dis. 2017; 228:60-67.
5. Brislin RW. Comparative research methodology: A cross-cultural studies. Int J Psychol. (1976); 11(3):215-229.
6. Lett J. Emics and ethics: Notes on the epistemology of anthropology. In Thomas N. Headland, Kenneth L. Pike & Marvin Harris (Eds), Emics and ethics: The insider/outsider debate. Newbury Park, CA: Sage. 1990; pp.127-142.
7. Olive J. Reflecting on the tensions between -emic and -etic perspectives in life history research: Lessons learned. Forum: Qualitative Social Research. 2014:15(2).
8. Essau CA, Qadir F, Maqsood A. Factor structure of the Urdu version of the Strengths and Difficulties Questionnaire in Pakistani adolescents. Acta Psychopathol. 2017; 3(1).
9. Hassan B, Vignoles V, Schwartz S. Researching identity styles in Pakistan: Confirmatory factor analysis and associations with commitment and value priorities. Identity. Int J Theory Res. 2018; 18(3):232-246.
10. Ayub N. Measuring hopelessness and life orientation in Pakistani adolescents. Crisis. 2009; 30(3):153-60.
11. Arafat SM, Rahman CH, Shalahuddin Q, Hafez M. Cross cultural adaptation and psychometric validation of research instruments: a methodological review. Journal of Behavioral Health. 2016; 5:129-136.
12. Borsa J, Damasio B, Bandeira D. Cross-cultural adaptation and validation of psychological instruments: Some considerations. Paidéia (Ribeirão Preto). 2012; 22:10.
13. Cassepp-Borges V, Balbinotti MAA, Teodoro MLM. Tradução e validação de conteúdo: Uma proposta para a adaptação de instrumentos. In L. Pasquali, Instrumentação psicológica: Fundamentos e práticas. Porto Alegre: Artmed. 2010:pp. 506-520.
14. Donbaek DF, Elklit A. A validation of the Experiences in Close Relationships-Relationship Structures scale (ECR-RS) in adolescents. Attachment & Human Development. 2014; 16(1):58-76.
15. Armsden GC, Greenberg MT. The inventory of parent and peer attachment: Relationships to well-being in adolescence. J Youth Adol. 1987; 16(5):427-454.
16. Parker G, Tupling H, Brown LB. A parental bonding instrument. British Journal of Medical Psychology. 1979; 52:1-10.
17. Rosenthal L, Kobak R. Assessing adolescents' attachment hierarchies: Differences across developmental periods and associations with individual adaptation. J Res Adol. 2010; 20(3):678-706.
18. Brennan KA, Clark CL, Shaver PR. Self-report measurement of adult attachment: An integrative overview. In J. A. Simpson & W. S. Rholes (Eds), Attachment theory and close relationships. New York: Guilford, 1998:(pp. 46–76).
19. Wei M, Russell W, Mallinckrodt B, Vogel L. The Experiences in Close Relationship Scale (ECR)-Short Form: Reliability, validity, and factor structure. J Personal Assess. 2007; 88(2):187–204.
20. Currie C, Roberts C, Morgan A, Smith R, Settertobulte W, Samdal O, et al. Young People’s Health in Context. Health behaviour in school-aged children: A WHO Cross-National Collaborative Study (HBSC International Report from the 2001/02 Survey). Health Policy for Children and Adolescents, No. 4 (World Health Organization), 2004.
21. Boyce W, Torsheim T, Currie C, Zambon A. The family affluence scale as a measure of national wealth: Validation of an adolescent self-report measure. Social Indicators Research. 2006; 78; 473-487.
22. Bartholomew K, Horowitz LM. Attachment styles among young adults: A test of a four category model. J Personal Social Psychol. 1991; 61:226-244.
23. Pervaiz Z, Sohail T. Breakfast, attachment styles and health of female university students. Int J Res Stud Psychol. 2014; 3(3):67-85.
24. Adil A, Kamal A. Relationship between love styles, attachment styles, and narcissism among university students. Indian Psychological Review. 2005; 64(1):33-52.
25. Schmitt DP, Alcalay L, Allensworth M, Allik J. Patterns and universals of adult romantic attachment across 62 cultural regions. J Cross-Cultural Psychol. 2004; 35:367-402.
26. Zigmond S, Snaith P. The Hospital Anxiety and Depression Scale. Acta Psychiatrica Scandinavia. 1983; 67:361–370.
27. Bjelland I, Dahl A, Haug T. The validity of the hospital anxiety and depression scale. An updated literature review. J Psychosomatic Res. 2002; 52(2):69-77.
28. Mumford IAK, Tareen MAZ, Bajwa MR, Bhatti R Karim. The translation and evaluation of an Urdu version of the Hospital Anxiety and Depression Scale. Acta Psychiatrica Scandinavica. 1991; 83(2):81-8.
29. Rosenberg M. Society and adolescent self-image. Princeton University Press, New Jersey, USA. 1965; pp.326.
30. Rizwan M, Malik S, Malik JN, Siddiqui SR. Urdu Rosenberg Self-Esteem Scale: An Analysis of Reliability and Validity in Pakistan. Sociology Int J. 2017; 1(2):00010.
31. Ziegler M, Bensch D. Lost in Translation: Thoughts Regarding the Translation of Existing Psychological Measures Into Other Languages. Euro J Psychol Assess. 2013; 29(2):81.
32. Muthén LK, Muthén BO. Mplus User’s Guide. Seventh Edition. Los Angeles, CA: Muthén & Muthén, 1998-2012.
33. George D, Mallery P. SPSS for Windows step by step: A simple guide and reference. 11.0 update (4th Edn). Boston, MA: Allyn & Bacon, 2003.
34. Hu L-T, Bentler PM. Cut-off criteria for fit indexes in covariance structure analysis: Conventional criteria versus new alternatives. Structural Equation Modeling. 1999; 6:1-55.
35. Kim Y. Statistical notes for clinical researchers: assessing normal distribution (2) using skewness and kurtosis. Restorative Dentistry & Endodontics. 2013; 38(1):52-54.
36. Finney J. A comparison of the psychometric properties of negatively and positively worded questionnaire items. ETD collection for University of Nebraska - Lincoln. 2001; AAI3009716.