We found considerable disagreements amongst consultants in virtual clinic. These discrepancies were more likely the greater the difference in levels of experience separating consultants and when mean visual field was abnormal and intraocular pressure was raised.

Glaucoma is a degenerative optic neuropathy that may be associated with significant visual impairment and compromised quality of life [1,2].

At present the prevalence of glaucoma is approximately 3-5% of people over 40 years of age, rising to almost 10% in people older than 75 years and representing 10% of UK blindness registrations [3,4]. Almost half a million people are currently affected by primary open angle glaucoma in England and that translates to over a million glaucoma related outpatient visits in hospital eye services (HES) annually with corresponding financial implications [5]. Recent recommendations on open angle glaucoma management by the National Institute of Clinical Excellence (NICE), coupled with the modifications in referral criteria from the Association of Optometrists have increased further the influx of new glaucoma referrals to HES [5,6]. Consequently, HES were challenged by this increased influx of patients, whilst also trying to cope with a 9% yearly increase in referrals due to an ageing population [6,7].

The National Patient Safety Agency has reported failures of the HES in providing access to care due to repetitive delays in hospital appointments, leading to irreversible visual loss [8]. Hence, innovation is required to optimally use the human and financial resources, so as to achieve timely access to treatment, whilst coping with an increasing workload. The Virtual Glaucoma Clinic (VGC) is a relatively new screening and diagnostic tool to manage glaucoma patients at a distance [9-11]. Clinical information, ocular imaging and diagnostic tests e.g. visual fields (VF), are delivered to an ophthalmic specialist for assessment without the need for a face-to face consultation. The benefits of VGC include increased access to specialized care for glaucoma, reduced cycle time (time from registration to departure from clinic), convenience and decreased absence from work and patient costs [12-14].

In 2012, Manchester Royal Eye Hospital (MREH) introduced a virtual follow up clinic to operate alongside the two existing higher tier services known as Optometrist Led Glaucoma Assessment Service (OLGA) and consultant-led outpatient clinics. All new referrals and complicated cases are assessed in either a consultant-led outpatient clinic or OLGA. The role of VGC is to monitor ocular hypertensives (intraocular pressure (IOP) >21 mmHg without glaucomatous optic neuropathy or visual field defect), glaucoma suspect (suspicious optic disc changes or visual field defect with or without raised IOP) and stable open angle glaucoma on topical mono-therapy (IOP deemed to be within target over 2 consecutive visits without prior glaucoma surgery). The virtual assessment involves integration of information from previous face-to-face consultations and results of diagnostic tests such as visual fields and optic nerve imaging. The grading specialist then decides on the medical treatment and follow-up care for the patient.

To date, various studies have examined the agreement amongst consultant led, optometry led and teleglaucoma clinics on the diagnosis and management of glaucoma patients [15,16]. No studies have formally investigated inter-consultant agreement on individual aspects of management outcomes in a virtual clinic environment. The purpose of this study is to evaluate inter-examiner variations in glaucoma management amongst consultants with different levels of experience, in a virtual clinic setting.

Three glaucoma specialist consultants, working at MREH, carried out the study. Based on the consultants’ experience in glaucoma management, they were labeled as senior, junior or new, when they had over 10-years, 5-years or 1-year experience respectively. On a prospective basis, within a 6-week period, between May and June of 2014, 120 consecutive patients were recruited into this study. Eight cases had to be excluded due to unobtainable data (e.g. missing tests or case notes) giving 112 patients in the study. All patients had previously attended MREH clinics and were attending for their follow up visits. The diagnosis from the last clinical entry and of the eye with a more severe diagnosis was used for analysis.

During the VGC, all patients underwent a series of predefined tests and examinations by a trained ophthalmic practitioner. Visual acuity (VA) and Goldmann intraocular pressure measurements were documented on a proforma in the patients’ hospital records. Visual fields (Humphrey Visual Field Analyser, 24-2 SITA standard), digital optic nerve imaging and an optical coherence tomography of the retinal nerve fiber layer (Topcon 3D-OCT 2000) were obtained and uploaded into our secure diagnostic imaging system. All 3 consultants then independently assessed each case record and the corresponding test results. The management outcomes would be documented in a standard proforma. Each consultant was masked from the decision made of the others. The decision from the consultant who was ultimately in charge of that patient’s glaucoma care would be used for the actual clinical outcome. Such outcome is communicated through to the patients via clinical letters. The decisions from the other 2 consultants were used for comparison only. The following management aspects were analyzed:

Time interval for follow-up appointment (Recorded in monthly unit); we considered agreement amongst the three consultants when the difference in recall time did not exceed 3 months.

Review place for next appointment; according to the severity of glaucoma findings, there were four possible management outcomes:

• Consultant-led outpatient clinic

• OLGA

• VGC

• Discharge to community optometrists.

Treatment plan; the management decisions included simple observation, commencement of topical treatment and change of treatment or cessation of treatment.

Statistical analysis

The management outcomes of all 3 consultants were pooled and entered on an MS Excel database. Statistical analysis was undertaken using SPSS v21.0 (SPSS, Chicago, IL, USA). Kappa coefficient was used to assess agreement on the above-mentioned variables. κ values above 0.81 illustrate excellent strength of agreement, between 0.61-0.80 illustrate substantial agreement, between 0.41-0.60 illustrate moderate agreement, between 0.21-0.40 illustrate fair agreement and between 0-0.21 illustrate slight agreement [17].

The cohort was classified into two groups. The first group included patients in whom all three consultants agreed on management outcomes. The second group consisted of patients in whom at least one consultant disagreed on management outcomes. Independent student-t- test was used to calculate the difference in mean values of age, VA, IOP, cup-to-disc ratio and VFMD between both groups for each aspect of management outcome. All data are presented as mean ± standard deviation unless otherwise indicated. A p-value of <0.05 was considered statistically significant for the purpose of our analysis.

Of the 112 patients recruited in this study, the proportion of glaucoma suspect was the highest. Figure 1 illustrates the percentage of patients for each diagnosis.

Figure 1: Pie Chart representing percentage of patients with glaucoma suspect, ocular hypertension and open angle glaucoma.

The demographics and clinical characteristics are summarized in Table 1. The eye with the most severe diagnosis was used to represent the mean values of these parameters.

Table 1: Demographics and clinical characteristics

Management outcomes analysis

Time interval for follow-up appointment: The level of agreement between the senior and the junior consultant was fair (κ 0.30). In comparison, the level of agreement between the junior and the newly appointed consultant was lower (κ 0.24). The lowest level of agreement was between the senior and the newly appointed consultant (κ 0.14). The overall percentage agreement amongst the 3 consultants on recall time for follow-up appointment was 25%.

Review place for the next appointment: The number of patients discharged from virtual clinic or referred to another service was comparable amongst the newly appointed and the senior consultant. The junior consultant discharged the highest number of patients. The overall percentage agreement was moderate (45%). Kappa agreement between any 2 consultants was fair, ranging from 0.22 to 0.36. The lowest level of agreement was again found between the senior and the newly appointed consultant (κ 0.22). A significant discrepancy was noted in one patient who would have been discharged by the new and junior consultant, whilst the senior consultant would have sent the patient to OPD for a Magnetic resonance imaging scan to investigate the cause of a decrease in VA.

Treatment plan: Kappa agreement between any 2 consultants for treatment plan ranged from 0.17 to 0.24 with highest level of agreement noted between the junior and the new consultant. The lowest level of agreement (κ 0.17) was between the junior and the senior consultant. The overall percentage agreement on treatment plan was excellent (85.7%).

Overall agreement on management outcomes for each condition

The highest level of agreement was noted in the treatment plan and review place of ocular hypertension (88% and 57% respectively). The lowest level of agreement was noted in the recall time and review place of glaucoma suspect patients (22% and 37% respectively). The level of agreements between glaucoma and ocular hypertension patients were comparable on all management outcomes except on treatment plan, where the level of agreement was 70% vs . 88% respectively.

Difference in mean value of patients’ characteristics in overall agreement subgroup against disagreement subgroup

Table 2, Table 3 and Table 4 summarize the clinical characteristics in overall agreement group compared to that in disagreement group on review time, review place and treatment plan respectively.

Table 2: Comparison of patients’ characteristics in agreement group vs. disagreement group regarding time interval for follow-up appointment.

Table 3: Comparison of patients’ characteristics in agreement group vs. disagreement group regarding review place for the next appointment.

Table 4: Comparison of patients’ characteristics in agreement group vs. disagreement group regarding treatment plan.

Virtual assessment clinics have attracted much interest in the management of glaucoma. Various studies have evaluated the diagnostic accuracy, the efficiency and the feasibility of implementing virtual clinic into HES. The efficiency, cost-effectiveness and safety of virtual glaucoma clinic are well reported in the literature [18,19]. Although data on patients’ satisfaction have been published, we found very limited data on the standard of care provided in virtual glaucoma clinic assessment [10,20]. This led us to conduct a study to shed light on the uniformity of care in a relatively innovative service. We believe this uniformity is important especially in a virtual service where high volume of patients is seen, often by different glaucoma specialists. Disagreement amongst the specialists may lead to confusion in the management plan for the patients and potential waste of resources.

Our study aimed at evaluating inter-examiner variations in glaucoma management amongst consultants with different levels of experience, in a VGC setting. The results demonstrate that the overall agreement amongst the 3 consultants on recall time for follow-up appointment was low (25%). With regards to review place and treatment plan, the overall agreement was largely superior (45% and 85.7% respectively). This may not be surprising because the recommended review intervals as suggested by European Glaucoma Society Guidelines and NICE guidance are often wide [5,21]. In the absence of clear local guidelines and the presence of a plethora of clinical suggestions in the literature, management plans are based on a variety of factors. Different years of experience result in differences in practices, influences and levels of familiarity with technology, which in turn lead to differences in clinical management. The latter became evident in our study also. It was clearly demonstrated that the level of agreement between the 2 consultants with greatest difference in levels of experience was low. Kappa agreement between the senior consultant and the newly appointed consultant was lowest for recall time and review place (κ 0.14 and κ 0.22 respectively). The only exception was found in treatment plan where we recorded the lowest kappa coefficient between the senior consultant and the junior consultant (κ 0.17). It is important to highlight that the best level of agreement on treatment plan was between the newly appointed consultant and the junior consultant (κ 0.24).

The highest level of agreement was reported for treatment plan and review place of patients with ocular hypertension (88% and 57% respectively). This is likely a reflection of adherence to the published NICE guidance for the diagnosis and management of chronic open-angle glaucoma and ocular hypertension in the UK [5]. Conversely we found the greatest level of disagreement amongst the glaucoma suspects which is the majority of our patients. This would explain the overall poor agreement amongst the 3 consultant in this study. Moreover, our findings show that consistency in management decisions is associated with patient’s characteristics. Consultants were more likely to disagree on review place for next appointment amongst patients with more abnormal mean visual field defect (-1.03 dB vs. -2.46 dB p=0.009, Table 3). The interpretation of visual field test can be variable and subjective while the test itself is variable and littered with false errors and fixation loss. Clinicians must decide if the visual field test is abnormal AND glaucomatous (especially for the glaucoma suspect which is the majority in this study) but also if the field is progressing. Hence it is not surprising to find the agreement in this study to be poor when the VFMD started to deviate from normal. We also found a tendency for consultants to disagree on treatment plan amongst patients with higher IOP (19.3 mmHg vs. 21.7 mmHg p=0.029, Table 4). This is likely to reflect on the lack of clear agreement or, more likely, documentation of target IOP in each patient case note. This in our view would increase outcome agreement.

The overall discrepancies in consultants’ decisions amongst patients affect uniformity of care. On the other hand we have no evidence from this study to suggest that this variability of management outcome has led to any harm. We appreciate that there is often more than one correct management plan for a specific patient. Our clinicians fail to agree most on glaucoma suspects, who are the majority in our study and indeed our daily work load. A more uniform approach to their follow up would benefit service planning and better utilization of resources. For example a standardized guidance on patient discharge would help to free up capacity and provide clearer instructions for the community optometrists to continue the care.

We appreciate there are limitations to our study. Firstly a larger number of patients would provide better comparison but was difficult to achieve due to time and work constraint. Secondly although the examiners were blinded from each other’s findings, they were all aware they were taking part in a study and who they were being compared to. Also decisions made on patient’s care in a study setting may not exactly reflect the decision made if it was the real virtual consultation. However, such bias is uniform across all three consultants. That may explain the significant discrepancy noted in the management of a few of our patients. Lastly we have studied the variability of management outcomes amongst consultant but we have not evaluated exactly why such disagreement exists; does consultant disagree most on VF or OCT? Further study would be required to evaluate which component of the diagnostic tests within the virtual clinic causes the most disagreement.

To the best of our knowledge this is the first study to evaluate inter-examiner variations in glaucoma management amongst consultants with different levels of experience, in a virtual clinic setting. Our study calls a need for the implementation of standardized guidance in virtual clinic service to aid clinicians in the formulation of more consistent management decisions to improve uniformity of care and service delivery.

The authors would like to thank Mr Robert Taylor, Dr Patricia Laurence and Dr Shelby Ramsamy for their comments on early versions of the manuscript.

The authors had no conflicts of interest to declare in relation to this article.

The following authors were involved; study design and planning (ECCTSK, EN, LA); conduction of the study (ECCTSK, EN, FS, LA); data collection and management (ECCTSK, JL, EN, FS, LA); data analysis and interpretation (ECCTSK, EN, LA); drafting the manuscript (ECCTSK, EN, LA) and the review and final approval of the manuscript (ECCTSK, JL, EN, FS, LA).