Self-report questionnaires based on clinical symptoms of posttraumatic stress disorder (PTSD), such as the Trauma Screening Questionnaire (TSQ) [1] are traditionally used in assessment settings as a mean to measure PTSD symptoms [2-5]. Moreover, in longitudinal studies or to evaluate treatment efficacy, clinicians consider changes in PTSD symptoms as a quantitative construct, and therefore use self-report questionnaires as measurement tools [6]. The present study examines what can be learned from test-retest TSQ data when psychometric evidence is lacking for the interpretation of the data as an effect of latent, one-dimensional, variability. If psychometric evidence supports the idea that a TSQ pattern of clinical symptoms can measure a reflective construct [7], it is possible to interpret test scores as tentative, ordinal measurements of that construct [8]. Thus, such psychometric framework may be useful to compare contrasted groups in light of how the construct is operationalized by the psychometric model. However, without supporting psychometric evidence, it cannot be stated that TSQ data or scores are useful in assessing the course of PTSD since there is no corroborated probabilistic evidence demonstrating that patterns of clinical symptoms and their associated test scores are able to reflect latent, one-dimensional, individual differences.

Without supporting psychometric evidence, a TSQ score is useful for counting the number of present symptoms reported by a given person at a given point in time. In the present study, the TSQ was used as a descriptive measure to assess that trauma experienced by people who lived through the January 12, 2010 earthquake in Haiti1. An earthquake that made about 222,000 dead, over 300,000 wounded, more than 4,000 to 7,000 amputees, 105,000 homes destroyed and 208,000 houses damaged [9] and survivors that present a high prevalence of PTSD and associated disorders [10-13]. Approximately half of the sample received psychological support based on the EMDR technique (Eye Movement Desensitization and Reprocessing) [14], administered in Haiti’s Psychotrauma Center. We were interested in comparing treated vs. non-treated groups on the basis of TSQ scores gathered within a testintervening- retest design. Given the test-retest scores, the proportion of people who reported fewer symptoms at retest in both groups-i.e., the treated and the non-treated ones-can be identified, and score proportions can be compared in order to determine the effects of the EMDR technique. However, due to the lack of suitable psychometric evidence, difference scores of the TSQ cannot be considered as measures of latent change. Hence, the problem raised by difference scores is that it is unclear whether decreased symptomology at retest is the result of a favorable change, since it is possible that participants reported new symptoms at retest.

An example

The TSQ is comprised of 10 binary clinical symptoms (Appendix) that describe people’s psychological state at a given point in time in a finite set of 210 possible observable states. Both states represent either the absence or the presence of the 10 signs, viz., a 10-tuple of 0s and 1s. Thus, the score associated with a state reflects the number of reported symptoms For example, a person who obtained a score of 8 at test (t₁) and the score of 5 at retest (t₂) signifies that eight symptoms were presents at t₁, while five signs were present at t₂. However, does this necessarily mean that there is an actual decrease in the person’s psychological state?

We refer to mathematical psychology for the concept of direct product order [15] to define the notions of unidirectional vs. bidirectional change of a 10-variate state. In this perspective, it is possible to define improvement and deterioration as unidirectional changes. Otherwise, the empirical meaning of improvement or deterioration is vague, that is, the difference score can refer to a number of possible changes in psychological state, such as the apparition of new symptoms.

A change in state is unidirectional if all the changes at the indicator level point in the same direction, i.e., from 0 to 1 (deterioration), or from 1 to 0 (improvement). A change in state is bidirectional if at least two indicators change in opposite directions. Thus, if bidirectional change occurred, it becomes problematic to interpret in terms of improvement or deterioration since the both kinds of phenomena, i.e., deterioration and improvement, occurred simultaneously.

To illustrate the interpretation problem posed by a difference of two test scores, let us further examine the example of the ordered pair (8, 5). The 8 point score indicates that two signs are absent at t₁. Thus, three classes of state transition are possible. (i) Both symptoms remain absent at t₂, in which case no new symptom has appeared, and three of the eight symptoms present at t₁ were no longer present at t₂: the change is unidirectional (Figure 1). (ii) One of the two symptoms absent at t₁ appeared at t₂: the change is bidirectional because deterioration occurred for one symptom while improvement occurred on four (Figure 1). (iii) Finally, both symptoms absent at t₁ appeared at t₂: the change is also bidirectional because deterioration occurred for two symptoms while improvement occurred for five (Figure 1).

Figure 1: Three classes of observed changes with 10 binary items yielding scores 8 and 5. Gray and white cases represent self-rated presence and absence of the corresponding clinical signs.

From the perspective of the test’s reference system, the usual interpretation of difference scores poses a validity problem [16,17]. In the present paper, we detail the logical conditions necessary for a clear understanding of improvement versus deterioration of an observable state, and address the empirical issue of assessing the extent to which, in a given dataset, it is possible to speak of strict improvement cases.

The concept of direct product order applied to the TSQ

Let M10 represent the set of possible 10-tuples associated with the TSQ indicators—the reference system of the test—, and let x=x₁ x₁ …x₁₀ and y=y₁ y₂ …y₁₀ denote the 10-tuples observed at t₁ and t₂, respectively. Let the ordering ≼ be defined on M10 as follows [14]:

y≼x iff ∀ i∈ {1,2,…,10},y_i ≤ x_i,      (1)

with x_i, y_i ∈ {0,1}. For example, consider x = 0011111111 and y = 0000011111; the statement y≼x is true because

     (2)

Now, let us consider the following incomparability case, where the score of 8 is associated with x = 0011111111 and the score of 5 is associated with y = 1000001111. The statement y ≼ x is false because the statement y₁ ≤ x₁ is false. In addition, the statement x ≼ y is false as well because the statement x₃ ≤ y₃ is false. Let us denote these two facts y≺≻x (and say that x et y are incomparable).

When comparing two 10-tuples x and y, there are six possible, mutually exclusive, possibilities:

1. y≺x: all symptoms are absent at t2 and present at t1 (unidirectional change, improvement);

2. y≻x: all symptoms are present at t2 and absent at t1 (unidirectional change, deterioration);

3. y∼x: ∀ i,x_i=y_i (identity, no change);

4. y≼x⋅[(y⊀x)+(y≁x)], where. and + represent the logical and and or, respectively: at least one symptom present at t1 disappeared at t2 without any other symptoms appearing (unidirectional change, improvement);

5. y≽x⋅[(y⊁x)+(y≁x)]: at least one symptom absent at t1 appeared at t2 without any other symptoms disappeared (unidirectional change, deterioration);

6. y≺≻x: bidirectional change, incomparability.

Research objectives

The following analyses were applied to a dataset of two groups of participants who experienced the earthquake of January 12, 2010, in Haiti. Participants in Group A followed EMDR sessions, while participants in Group B did not receive any treatment.

Rasch model test: The Rasch model is a suitable psychometric model that can be applied in order to explain the probabilistic effect of latent variability on a latent continuum in an observed dataset [18]. But since psychometric models aren’t ironclad [19,20] we hypothesized that the Rasch model would be rejected as a proper null hypothesis to account for the data, which entails that it is necessary to rely on an alternative measurement framework if one wants to assess the proportion of participants whose observed state improved from test to retest.

Interpreting changes in 10-tuples: Considering the matrix of the 220 possible state changes, there are 863,820 possible variations for which the corresponding difference scores are non-null (a spreadsheet is available on request to the author). Among these changes, 116,050 are unidirectional, which represents about 13% of possible configurations in changes. In probabilistic terms, if the observed changes occur according to a uniform random law, the proportion of the observed unidirectional changes should not deviate significantly from 0.13. Rejecting this null hypothesis would offer some empirical support to the idea that, at the population level, the phenomenon of unidirectional change obeys some structural constraints based on the principle of unidirectional change, even if a substantial proportion of bidirectional changes may be observed (and thus interpreted as unexplained data).

Furthermore, it is of interest to compare the proportions of improved cases in the two groups with respect to the observed cases of unidirectional changes, as practitioners expect that the EMDR technique is beneficial to people facing psychological distress.

Participants, design, and materials

The sample comprised 372 adults divided into two groups: Group A, with a mean age of 35.36 years (SD = 12.71 years), consisted of 169 adults (75 men) who received EMDR treatment between the test and the retest Participants at the center of psychotrauma of Haiti which was established in Port-au-Prince after the earthquake of January 12, 2010 and consists of twenty psychologists; and Group B, which had a mean age of 24.41 years (SD = 9.21 years), and consisted of 203 adults (104 men) none of which received EMDR treatment between the test and the retest. Three inclusion criteria were applied for the recruitment of participants: (i) being at least 16 years old, (ii) having experienced the earthquake in Haiti; (iii) in regard to participants in Group B, not having received EMDR treatment between the test and the retest; (iv) they had to have signed the informed consent form. The Institutional Review Boards of Ministry of Public Health and Population, Haiti Psychotrauma Center and the State University of Haiti approved the protocol of research and all recruitment and assessment procedures of this study.

The data of Group A came from the Haiti Psychotrauma Center. Group A’s participants were assessed before and after completing eight to twelve EMDR sessions, within a period varying from 12 to 19 days with an average time lapse of 17.09 days (SD = 2 days). The EMDR sessions consisted of a brief therapy widely used in the treatment of PTSD following a standardized procedure [14,21]. EMDR treatment offered included assessment, relaxation, desensitization, installation, body scan and reevaluation.

Participants in Group B were selected according to the same socioprofessional categories as that of Group A. Participants of Group B were assessed within a period varying from 18 to 20 days with an average of 19.01 days (SD = 0.82 day). At the test, 339 people were assessed (103 people living in one of the three camps set up in the capital, 68 students and pupils of at least 16 years, 39 members of a women’s organization, 67 executives from different institutions of the capital--including 18 bank employees--and 62 other people). At the retest, 232 of were assessed once again.

The self-report clinical psychological state of participants was assessed using the TSQ. Brewin et al. [1] based the 10 items of the TSQ on the Post-Traumatic Stress Disorder Symptom Scale Self Report (PSSSR) [22]. The items correspond to the diagnostic criteria B (intrusive recollection) and D (hyper-arousal) as listed in Post-Traumatic Stress Disorder section of the DSM IV [23].The first author of this article adapted the questionnaire items in Creole to help respondents better understand the meaning of each item if needed.

Each participant had to state if he had in relation to the traumatic event (the earthquake of January 12, 2010) the reactions described in the questionnaire: “please indicate (Yes/No) whether or not you have experienced any of the following at least twice in the past week”. The same instructions were used for the retest.

Analyses

The Rasch model was estimated using the computer software WINMIRA [24]. As the explicative latent variable requires no distributional assumption, estimation was based on the whole dataset (test and retest data of both groups). We assessed goodness of fit with the Pearson chi-square and the Cressie-Read statistic obtained from 300 bootstrap samples, as recommended by von Davier [24].

Concerning the analyses based on the direct product order, the testretest comparisons of the 10-tuples were computed for each group with a suitable JAVA computer program. Within each group, the test-retest pairs were matched to one of the six relational classes defined in the Introduction section.

After having removed missing data, Groups A and B comprised 169 and 203 test-retest cases, respectively. As expected, the Rasch model did not fit the data, the Cressie Read and Pearson χ² statistics being highly significant (p < 0.001).

The average numbers of self-reported clinical symptoms at the test were similar, t = 1.65, p = 0.10. Hence, differences in proportions of improved cases cannot be explained by an average difference between the two groups at the test.

Table 1 shows that Groups A and B contain 88 and 71 unidirectional change cases over 137 and 133 change cases, such that the corresponding difference scores are non-null respectively, which yields the proportions of 0.64, CI_0.99 (99% confidence interval, normal approximation) = [0.54, 0.75], and .56, CI_0.99 = [0.45, 0.67], respectively. Hence, the null hypothesis according to which the probability of unidirectional change is about 0.13 was rejected. The probability of unidirectional change in groups of respondents is clearly higher than that which could be expected from a mere uniform random law in the test-retest reference system.

Table 1: Distributions of the development of PTSD among the participants of the group A (N = 169) and the group B (N = 203) / comparisons between t₁ and t₂.

The proportions of improved cases in the two groups with respect to the observed cases of unidirectional changes were pA = 0.79 and pB = 0.37, z = 6.29, p <0.0001, Cohen’s h = 0.89. If the proportions refer to the number of cases that exhibit non-null difference scores, the effect size was smaller, p’A = 0.59 and p’B = 0.33, z = 4.29, p < 0.0001, Cohen’s h = 0.53. Thus, the use of the subsamples of unidirectional changes is useful to exhibit a strong contrast between the two groups.

The main objective of the present study was to propose and apply a well-defined conceptualization of psychological change using the TSQ among people who experienced the January 12, 2010 earthquake in Haiti. Since the data came from a population of people who received or not psychological treatment based on the EMDR technique, the secondary goal was to compare the proportions of improved cases in the two groups.

The Rasch model did not fit the data, therefore it could not be retained in the interpretation of TSQ scores as being more or less imprecise measurements that refer to a single continuum representing quantitative differences in the intensity of PTSD (along with local independence). Alternatively, the direct product order measurement framework can be applied. In this perspective, bidirectional change represents approximately 40% of all the cases. In practical terms, this result shows that incomparability occurs in significantly high proportions. Therefore, because or the lack of scientific evidence for a quantitative interpretation of the scores and the multivariate nature of psychological states, practitioners must be aware that referring to an individuals’ psychological state in terms of improving or deteriorating can be, at best, vague, and, at worst, misleading.

On a positive note, having such a substantial proportion of unidirectional change cases is not only useful to compare clinical groups, but also in demonstrating the benefit of following an EMDR therapy. How can such findings be interpreted from a clinical perspective?

First, if researchers critically examined the construct validity of PTSD [25], the present study emphasizes the necessity to refer data in their empirical meaning.

Thus, a large amount of cases exhibit incomparable response patterns, and the language of improvement/deterioration of in reference to a psychological state is deemed inappropriate of one is concerned by a clear definition of improvement/deterioration. A practical consequence is that the clinician interested in assessing the course of a psychological state should be prepared to detect an incomparability issue. Further, research is needed to document more qualitatively such cases, by investigating the psychological meaning behind quantitative self-report responses.

More generally, self-report measures should not be used to identify clinical symptoms per se. Because self-report questionnaire require respondents to produce a standardized discourse about themselves, such an identification entails that self-raters are reliable when they describe themselves, which is an assumption that requires critical examination. In summary, findings showed that the proportion of people who described themselves twice in such a way that their descriptions follow the principle of unidirectional multivariate change is higher than chance, even if about 40% of people provide bidirectional descriptions, suggesting that bidirectional change deserves further investigation. While the two groups exhibited the same average number of self-rated clinical symptoms at the test, the proportion of improvements is notably higher in people who followed an EMDR therapy.

These findings present some limitations. First, only one questionnaire was used in the present study. Even though it does not permit comparison between different instruments, this choice was made in order to focus on the novelty of this paper, being the first, to our knowledge, exploring this specific model. It also allowed for a more detailed explanation of the methodology used in this study rather than presenting results that are less related to the purpose of this paper. The TSQ was used primarily as an example serving the demonstration of the model that could be applied to other self-reported questionnaires in the field of trauma or other. The second limitation pertains to the fact that relatively few victims received psychological treatment after the earthquake. The few who have benefited from such professional support might present a recognition effect that could possibly influence their score. However, the main objective of this paper was to demonstrate that the traditional way to assess psychological evolution with general scores does not represent truthfully the reality, and this with or without the administration of a treatment. Thus, the relatively small number of participants having benefited a professional psychological support doesn’t not affect the conclusions on the initial hypothesis. The final limitation to this study is the fact that the results from the Rasch model were not presented. Being non-significant, we chose not to present these results in the paper. Besides, here again the main purpose of the study was to offer and explain a new method of analysis, while the Rasch model is now widely used in studies in psychiatry and psychology.

In conclusion, the results of this study showed that the direct product order method presented assesses, better than traditional psychometric methods, what is truly happening empirically in the referential of a psychological or psychopathological evaluation test, therefore the evolution of symptoms. It is thus a better tool for practitioners and researchers in the assessment of the evolution of the psychological state of clients and patients, following psychotherapy or even psychopharmacotherapy.