Acute abdominal pain accounts for almost 10% of all emergency department visits [1-3]. Acute appendicitis and acute diverticulitis are frequently suspected serious causes for acute abdominal pain [1-3]. Correctly identifying patients with acute appendicitis or acute diverticulitis is a diagnostic challenge [4-11]. Guidelines often suggest that both these diagnoses can be made based on history and physical examination [12,13]. Several studies though, have demonstrated that medical history and physical examination alone lead to misdiagnosis resulting in delayed or inadequate treatment [1,2,14]. Therefore most patients are referred for additional imaging [3,6,13]. However, only 50% of patients with clinical suspicion, who are referred for imaging, are ultimately diagnosed with acute appendicitis or diverticulitis [1]. This implies over-utilization of imaging modalities [6-15]. Additionally, approximately 90% of patients have uncomplicated diverticulitis and imaging may be omitted because it will not alter the management [6,12,13].

Over-utilization of additional imaging places an unnecessary burden on both patients and health care facilities [4,11,16-19]. Selective use of imaging in patients suspected of acute appendicitis and diverticulitis would improve both patient satisfaction and reduce health care costs. An international survey amongst emergency physicians demonstrated the need for a decision rule to select patients with acute abdominal pain for Computed Tomography (CT) [20]. Decision tools potentially can prevent over-utilization of imaging by selecting patients for diagnostic imaging [16,21-24]. A decision rule is comprised of several variables from medical history, physical examination and simple tests [25]. Decision rules can increase or decrease the pre- test probability of an outcome suggesting a course of action (such as imaging) [21]. Several decision rules have been developed to guide decision making in patients suspected of acute appendicitis or diverticulitis. The most commonly used decision rules are the appendicitis inflammatory response score (AIR) and the Alvarado score for appendicitis. For the diagnosis of diverticulitis the emergency department (ED) triad and the Clinical scoring system (CS tool) for diverticulitis have been introduced and validated recently [7,8,10,26-29].

Several factors contribute to successful implementation of decision tools into daily clinical practice [16,21-24,30]. First, as is the case for the mentioned decision tools, studies must have demonstrated sufficient diagnostic accuracy [7,26,28]. A second step is external validation [29,31]. External validation of the diverticulitis tool has shown a positive predictive value of 97%, and 3% false positives [29]. The triad has a sensitivity of 36%, meaning that the tool misses 64% of patients with diverticulitis. For appendicitis the external validation of the Appendicitis Inflammatory response (AIR) score has shown a positive predictive value of 79%, and 23.3% false positives [31]. A higher positive predictive value is seen for a score of more than 8, but this cut-off had sensitivity for appendicitis of a mere 10%. This means that the tool misses 90% of patients with appendicitis, but when the score is above 8 then the presence of appendicitis is highly probable.

Accuracy is one thing but usage in daily practice is another. The rationale for the use of decision tools lies in their ability to alter the daily diagnostic work-up. Therefore, the true effect of a decision tool on patient care should be assessed by a prospective implementation study before widespread implementation of decision tools can be advocated [21].

The objective of this study was to assess the influence of the use of decision tools for appendicitis and diverticulitis in clinical practice. The primary objective was to evaluate the influence of decision tools on the impact on the certainty of diagnosis in relation to initially requested diagnostic imaging. For appendicitis a new decision tool had been constructed that produced a more detailed probability (expressed as percentage) than the three categories (low, intermediate, high) of the AIR tool.

The AAP (Acute Abdominal Pain) study was a multi-center prospective cohort study conducted between December 2009 and January 2013. The objective of the AAP study was to evaluate the impact of decision tools on the certainty of diagnosis in relation to initially requested diagnostic imaging (according to present day practice). Approval of the institutional review board of each participating center was obtained prior to initiation. Three hospitals in the Netherlands participated in patient accrual: one academic hospital (Academic Medical Center, Amsterdam) and two large teaching hospitals (Onze Lieve Vrouwe Gasthuis and Sint Lucas Andreas Ziekenhuis, Amsterdam). Patients gave written informed consent before inclusion. Patients presenting at the emergency with non-traumatic abdominal pain with duration of more than two hours and less than five days, were considered eligible. Exclusion criteria were age under 18 years, hemorrhagic shock due to gastrointestinal bleeding, ruptured aortic aneurysm and known pregnancy. Patients in whom further diagnostic investigations such as imaging were already performed for acute abdominal pain were also excluded. Patient were either self-referred or referred by their general practitioner.

Decision tools

The decision tool for appendicitis was built using logistic regression analysis on a cohort of 422 (41%) patients with clinical suspicion of appendicitis from a large database containing 1021 patients with acute abdominal pain [1]. The prevalence of appendicitis in this cohort was 251 of 422 (59%). The incorporated variables were identified based on the literature [32]. The final appendicitis tool consisted of five variables; sex, migration of pain to the right lower quadrant, vomiting, rigidity of the right lower quadrant, and white blood cell count. A nomogram was created of the regression coefficients from the logistic regression model. Each of the variables was translated into points based on the coefficients in the final model, adding up to a total of 60 points. The achieved score in points was correlated with a probability (%) that the diagnosis of appendicitis was correct. The decision tool had an area under the operator receiver curve of 0.78 (95%CI 0.73-0.82). More detailed information on this decision tool is provided in appendix 1.

Appendix 1: Decision tool Appendicitis

The decision tool used for diverticulitis was the Emergency Department triad [7,29]. The ED triad consisted of three parameters: tenderness exclusively in the left lower quadrant on physical examination, absence of vomiting, and a C-reactive protein > 50 mg/L. The ED triad is conclusive for diverticulitis when all three parameters are positive. When not all parameters are positive the ED triad advises to perform additional imaging [7,29]. More detailed information on this decision tool is provided in appendix 2.

Appendix 2: Decision tool acute diverticulitis

Diagnostic protocol

Treating physicians at the emergency department prospectively identified eligible patients. The attending surgical residents prospectively registered history taking, physical examination and laboratory tests. Laboratory tests were requested upon judgment of the attending resident. Afterwards they were asked to give their clinical diagnosis and record their certainty of the diagnosis by means of a Visual Analogue Scale (VAS). The scale ranged from 1 to 10 with 10 indicating absolute certainty. Residents also recorded whether they would perform additional laboratory investigations, imaging investigations, whether they would consult another specialism, and their management plan. All data were recorded in an online structured case record form. After completion of this step the case record form could not be changed.

When the clinical diagnosis of the resident was acute appendicitis or diverticulitis the disease-specific decision tool was prompted automatically from the online case record form and had to be completed. When the decision tool was completed the outcome of the tool was provided to the residents. After review of the decision tool results the residents recorded their certainty of the diagnosis once more. Residents also answered questions about the role the decision tools played in that diagnostic process, such as their influence on the certainty of the diagnosis and their influence on the consecutive course of actions. After the appendicitis decision tool residents recorded whether the decision tool influenced their diagnosis and certainty of diagnosis. In case of the decision tool for diverticulitis residents recorded whether the tool had influenced their suggested utilization of imaging.

The outcome of the decision tool was not binding in this research setting. Initiation of further diagnostic work up was based on the overall judgment of the attending resident, and according to the standard practice Dutch guidelines for acute abdominal pain [33]. This guideline also elaborates on the need for additional imaging due to the limited accuracy of a clinical diagnosis in patients with acute abdominal pain presenting at the emergency department.

Reference standard

Additional data on clinical, laboratory and surgical findings as well as pathology results, imaging reports, outcomes of treatment and outpatient clinics information were collected for a follow-up period of three months. Data was summarized in a standardized manner including all available information collected during the first emergency department visit and the follow-up period.

Four expert panels of each two surgeons with extensive clinical experience assigned the final diagnosis based all information. Patient cases were divided among the expert panels. Cases were assigned to expert panel members who had not been involved in the initial examination or management of the case. Each member of an expert panel individually assigned a final diagnosis to every case; disagreements were resolved during consensus meetings with a third expert.

Statistical analysis

The diagnostic accuracy of the clinical diagnosis versus the clinical diagnosis in combination with the decision tool was calculated by comparison with the final diagnosis. The percentage of missed cases (1- sensitivity) and the percentage of false positive cases (1- positive predictive values) were also calculated. Difference in mean levels of certainty of diagnosis was tested using an independent sample t-test. All statistical analyses were performed with statistical analysis software (SPSS version 18.0; IBM, Armonk, NY).

A total of 294 patients were included. Acute appendicitis was suspected in 143 patients and acute diverticulitis in 20 patients based on clinical evaluation (Figure 1); 56% (91/163) of patients were females with a median age of 33 (IQR 24-49); males had a median age of 38 (IQR 29-48). The clinical diagnosis was in accordance with the final diagnosis in 56.6% of patients suspected of appendicitis (81/143) and in 55% of patients suspected of acute diverticulitis (11/20 patients).

Figure 1: Study flow chart

Nonspecific abdominal pain (NSAP) and bowel obstruction were the most common other final diagnoses (Table 1a and 1b).

Table 1a: Differential and final diagnoses in patients suspected of appendicitis (n=143)

Table 1b: Differential and final diagnoses in patients suspected of diverticulitis (n=20)

Diagnostic accuracy of clinical diagnosis vs. clinical diagnosis plus decision tool

Table 2 summarizes the diagnostic accuracy of clinical evaluation compared to the accuracy of the clinical evaluation combined with the decision tool in patients suspected of appendicitis. The clinical diagnosis was correct in 81 of the 143 patients (56.6%). In 43% of patients (62/143 patients) clinical evaluation alone would lead to a false positive diagnosis. . The combination of a clinical diagnosis and decision tool was registered in 132 patients suspected of appendicitis and correct in 72 patients (54.5%). In 40% of patients clinical evaluation combined with the decision tool would lead to a false positive diagnosis.

Table 2: Diagnostic accuracy of the clinical diagnosis compared with the clinical diagnosis plus the decision tool

In 55% of patients (11/20) suspected of diverticulitis the clinical diagnosis was correct. Clinical evaluation alone would lead to a false positive diagnosis in 45% of patients (9/20 patients). Due to the limited amount of patients with a final diagnosis of diverticulitis the accuracy of the clinical diagnosis combined with use of the ED triad was not calculated.

Utilization of hospital resources before application of decision tools

Table 3 depicts the utilization of hospital resources as suggested by residents after clinical evaluation.

Table 3: Suggested utilization of hospital resources before use of decision tools

Proposed utilization of hospital resources was recorded in 19 of the 20 patients suspected of diverticulitis and in 137 of 143 patients suspected of appendicitis.

Additional imaging was requested by residents in 142 of 156 patients (91%). Residents suggested need for abdominal ultrasonography in nearly all of these patients (88.5%). In only 0.06% of patient’s consultation of another specialism was requested as next step in the diagnostic work up. The most commonly suggested management plan after clinical evaluation (53.8% of patients) was admission for operative treatment. Outpatient re-evaluation (21.8%) and admission without treatment (17.3%) were second and third most suggested by residents.

Impact of decision tools on clinical practice

The mean level of certainty of the clinical diagnosis was higher 7.19 (IQR 6-8) in patients with a final diagnosis of appendicitis compared to patients with a final diagnosis that turned out to be other than appendicitis 6.5 (IQR 6-7). The mean level of certainty decreased to 7.08 (IQR 6-9) in patients with appendicitis after the use of the decision tool for appendicitis. In patients without acute appendicitis the mean level of certainty also decreased to 6.02 (IQR 6.02). In the majority of patients the level of certainty remained unchanged; in 51% of patients with a final diagnosis of acute appendicitis and in 49% of patients without appendicitis. The level of certainty of residents increased after completion of the decision tool for only 19.2% of patients with final diagnosis appendicitis and 13.6% patients without appendicitis. In 29.5% of patients with appendicitis the level of certainty even decreased after use of the decision tool, and it decreased in 37.3% of patients without appendicitis. In only 5.1% of patients with appendicitis residents described an influence of the decision tool on their assigned diagnosis, compared to 13.6% in patients without appendicitis. Their certainty of diagnosis was positively influenced in 24.4% of patients with appendicitis and 32.2% without appendicitis.

The mean level of certainty after clinical evaluation was higher (7.55 IQR 7-8) in patients with diverticulitis compared to patients with an alternative final diagnosis (5.22 IQR 5-6). The mean level of certainty after completion of the decision tool remained similar in patients with diverticulitis (7.55 IQR 7-9) but decreased to 5.13 (IQR 5-6) in patients without diverticulitis. In 36.4% of patients with diverticulitis and in 37.5% of patients without diverticulitis the level of certainty increased. In 36.4% of patients with diverticulitis the level of certainty decreased after the decision model compared to 25% in patients without diverticulitis. In the minority of patients the level of certainty remained unchanged; in 27.3% of patients with a final diagnosis of diverticulitis and in 37.5% of patients with another final diagnosis. In only 18% of patients with diverticulitis the decision tool was reported to influence the utilization of imaging, and in none of the patients with an alternative final diagnosis (Table 4).

Table 4: Impact of decision tools on the level of certainty of the diagnosis.

This multi-center prospective cohort study demonstrates that use of decision tools for acute appendicitis and diverticulitis has limited influence on the modest accuracy and certainty of a clinical diagnosis. Currently, decision tools are unlikely to influence utilization of hospital resources. The diverticulitis ED triad has some potential to influence daily practice. In only 18% of patients with diverticulitis the decision tool was reported to influence the utilization of imaging, and in none of the patients with an alternative final diagnosis.

The decision tools had limited beneficial effect on the certainty of diagnosis of the residents, in a setting where they were not consciously aware of the accuracy of the tool but had knowledge of the limited accuracy of a clinical diagnosis. The level of certainty decreased or remained unchanged in the majority of patients after use of the disease-specific tool. In suspected diverticulitis the level of certainty changed more often than in suspected appendicitis.

Over the past years decision rules have been extensively described in the literature [7,8,10,26,28,34]. However, most of these decision tools are seldom used in daily practice. Successful implementation into daily practice has been demonstrated in only a select few of these rules, such as the Ottawa ankle rules [23,35]. Adoption of decision tools by physicians is affected by attributes of the decision tool such as the accuracy and reliability. Several studies have demonstrated high accuracy and reliability of the emergency department triad for acute diverticulitis [7,29]. The low diagnostic accuracy of the appendicitis decision tool in this study influences the adoption of the decision tool in daily practice. The question remains whether another decision tool such as the AIR score would have performed better in terms of level of perceived certainty of the clinical diagnosis. The degree of dissemination of the decision tools in this study is limited, inherent to the design of the study that set out to evaluate the potential of the tools to change future daily practice. It has been demonstrated that physicians are less likely to change their behavior based on a decision tool that they are unfamiliar with [21,30,35,36]. Even though the Dutch guidelines for acute diverticulitis describe the emergency department triad and its diagnostic characteristics we are not sure how well disseminated the emergency department triad is [37]. The Dutch guideline for diagnostics in patients with acute abdominal pain had been recently introduced and that might have influenced the value that the residents attribute to the outcome of the decision tool.

Several other factors can influence the use and subsequent efficacy of a decision tool. The decision tool has to bring an advantage to current practice such as a higher diagnostic accuracy or proven economic benefits. Another important factor is the perceived easiness to remember and use the decision tool [25,36]. Decision rules consisting of more than 3 variables are often too difficult to memorize and are therefore less likely to be used in daily practice. Decision rules that provide a probability are also less likely to alter daily practice than decision rules that suggest a direct course of action. Implementation of any of the currently existing appendicitis decision tools might be hindered by several of these attributes. The most commonly used decision tools for appendicitis consist of more than three variables, result in a probability of appendicitis instead of suggesting a course of action or have insufficient discriminating capacity [8,10,26,31]. These factors diminish the chance of successful implementation of any of the current decision tools for appendicitis. In case of the emergency department triad these hindering factors are less prominent. The rule consists of only three variables which are easy to memorize, it suggests a course of action and its diagnostic accuracy has been demonstrated in several cohorts [7,29]. Importantly, residents value its results just as low as the decision tool for acute appendicitis.

The fundamental principle of a decision tool is to improve daily practice. Successfully implemented decision tools can alter daily practice in several ways; it can increase certainty in medical decision making by objectifying the interpretation of clinical data, it can decrease burden for patients and reduce hospital costs by preventing unnecessary imaging [16,21]. Although several studies have described the development and external validation of decision tools no study has prospectively studied the implementation of decision tools for patients suspected of acute appendicitis or diverticulitis.

Implementation studies are the golden standard to evaluate the clinical value of decision tools in ‘the real world’. By implementing decision tools in a patient series in multiple hospitals we were able to assess the impact of decision rules in daily practice in various settings [21,35]. Another advantage of this study design is that selection bias was prevented as the decision tool was automatically applied in all consecutive patients for whom the decision tool was developed. By recording residents’ certainty of diagnosis before and after use of the decision tool, we were able to assess whether they believed that the decision tool changed their diagnosis, the certainty of their diagnosis and the suggested utilization of imaging. All outcomes were self-reported outcomes by residents, which might introduce another source of bias. Self-reported behavior might not always be in accordance with actual behavior [36].

Based on these results we can conclude that currently both the decision tools for acute appendicitis and acute diverticulitis will have limited influence on diagnostic certainty and therefore on utilization of hospital resources. Successful implementation of decision tools is dependent of several factors. The Emergency Department triad has the potential to influence daily practice. It has a sufficient diagnostic accuracy, is easy to memorize and apply and provides the user with a suggested course of action. Future studies should therefore be aimed at assessing the hindering factors that prevent widespread implementation of the Emergency Department triad. However, our study only included a limited amount of patients suspected of acute diverticulitis. After widespread implementation of the Emergency Department triad the effectiveness in daily practice should be reevaluated in a larger population. The appendicitis decision tool has too many limiting attributes to be successfully implemented, but this also holds true for other tools. Future studies should aim at developing a simple decision model that is easy to incorporate into daily practice and suggests a clear course of action.