Objective assessment of surgical skills of trainees during a residency program is crucial in the education and development of the surgical skills of trainees. Methods of assessing the performance of the residents in the operating theatre is vital in order to achieve competency in technical skills and acquisition of surgical expertise ultimately ensuring best optimal patient safety [1]. Efforts have been made to design assessment tools aiming to evaluate surgical skills in the operating theater that meet the standards of feasibility, reliability and validity. The Objective Structured Assessment of Technical Skills (OSATS) was developed by Winckel et al. [1] to conduct a formal assessment of surgical resident’s technical skill, which was later modified by Martin et al. [2].

Numerous researches in the surgical literature has been able to develop OSATS-based assessment tools with proven feasibility, reliability and validity in different surgical disciplines [3-5]. In otolaryngology head and neck surgery, such objective tools have been successfully examined as being reliable and valid in different procedures such as mastoidectomy, sinus endoscopic surgery and laryngoscopy [6-8].

Thyroid surgery can be considered as an ideal representation of head and neck procedures in which surgical residents need to have integrated knowledge of the challenging anatomy and requires development of meticulous surgical technical skills, hence the demand for a structured observational assessment is undeniable [9]. Intending to address this issue, an OSATS-based instrument using Global Rating of Thyroid Surgical Skills (GRTSS) has been designed in this pilot study to assess the resident’s expertise in technique for thyroid surgery. The purpose of this pilot study was to test this tool’s feasibility, reliability and construct validly in evaluating otolaryngology head and neck surgery residents at various postgraduate levels.

After obtaining an approval from the ethical committee to carry out this prospective study, consent was obtained from residents at various levels of training and enrolled. The staff assessors were informed about the study and agreement was reached about their participation.

The design was a prospective, observational study within the operating theatre of a teaching hospital. Trainees were directly observed performing thyroidectomy.

A 2-page evaluation form was developed in conjunction with the OSATS evaluation form that was originally developed by Winckle et al and modified by Martin et al. [2,3] (Table 1) to assess the surgical skills of residents. A one to five (five = excellent) scale was used for evaluation.

Table 1: The global rating score (Martin et al.)[3].

The (GRTSS) evaluation instrument (Table 2) was designed with input from academic otolaryngologist, fellowship trained head and neck surgeons and experts in medical education. Their comments and suggestions were incorporated, establishing face and content validity. Finally 16 items of task performance were selected with a five-point Likert scale linked to specific descriptors. The number five in the scale was equivalent to the performance of a resident who can perform that step independently and safely.

Table 2: Global Rating for Thyroid Surgical Skills (GRTSS). Please mark the candidate’s performance in the above scale. Scale 5 is equivalent to a resident that can perform the step independently and safely.

The participants’ scores were calculated as percentages and were entered using SPSS (Statistical Package for the Social Sciences, version 17.0) for reliability and construct validity measures.

A total of 23 thyroid surgeries were performed by 7 residents at three different levels of postgraduate years; R3, R4 and R5. They were assessed by a single assessor using two different assessment tools the Global Rating Score (GRS) and Global Rating of Thyroid Surgical Skills (GRTSS) displayed in Tables 1 and 2 respectively.

The assessors reported that completing the assessment forms were convenient and not time consuming which is a major issue in an operation theater setting. The tools were also described as being simple, brief and to the point by the assessors. Based on these observations the instrument can be considered as feasible.

The interterm reliability was determined by the assessment tool’s internal consistency. It was evaluated with Cornbach’s α, which proved the tool’s high reliability, 0.707.

It has proven to be construct-valid as senior trainees achieved higher scores than junior trainees. This was measured comparing means using independent t sample test (P<0.001, mean difference: 11.375, 95% confidence interval 6.25-16.49). The GRS also showed statistical significance (P<0.005 mean difference: 6.958, 95% confidence interval 2.787-11.129).

Furthermore, to provide evidence of construct validity the a set of residents performing the surgeries were assessed by the same assessor in two separate training years, first in year 4 and next in year 5. The scores was compared between the trainees attending senior and junior years. Notable improvement was observed in the scores of the same trainees in senior year (Figure 1), proving the construct validity of the instrument. This was measured comparing means using independent t sample test (P<0.005, mean difference: 18.8, 95% confidence interval 7.45-30.14). However, the in this set, GRS did not show statistical significance (P<1.80 mean difference: 12.2 95% confidence interval 6.96-31.36).

Figure 1: Residency level means (scores in percentages) using Global Rating Score (GRS) and Global Rating of Thyroid Surgical Skills (GRTSS).

The emphasis on work-place assessment methods has been growing as measuring a trainee’s knowledge, skills, judgment or professional behavior is of determinative values in the educational or training programs [10]. Standardized methods of assessing the surgical skills of trainees in the operating theatre require valid and reliable instruments to ensure that graduates’ performance is in compliance with the accepted professional demands [10]. To address the growing need for a structured assessment tool to properly evaluate the operative skills of otolaryngology trainees/residents performing thyroid surgery this assessment tool was developed, which proved to be valid and reliable as well as acceptable to assessors. The feasibility and cost-effectiveness of the assessment tool developed in this study were present, as these are essential components for an assessment method to be convenient and for proper implementation of it in a training program.

Considering the reliability, the internal consistency was found to be significant when compared to the known global rating scale that was used as a second tool in this study.

Since measuring the progress of trainees over time is crucial in certifying competence, this instrument was able to demonstrate achievement of higher score with advancement in training years. It was also able to capture progress within the same participant after gaining more experience and knowledge with subsequent evaluations.

Limitations to this study include the small sample size, however, increase in the number of participants would be considered to confirm findings in future studies as this is a preliminary pilot one. Another limitation which too will be addressed in the subsequent larger scale study is that inter-rater agreement was not possible to be calculated since every participant was assessed by a single assessor.

The otolaryngology residents were directly observed performing thyroid surgeries on patients in presence of the assessors who were consultants, in contrast to other studies that were done at other different settings using bench models or live anaesthetized animals [2].

This tool’s components are broadly similar to those developed by Diaz Voss Varela et al. [11], however, this instrument has longer and separate items that would play an important role in accurate evaluation of operative skills of the residents. This study’s results were consistent with those developed by Diaz Voss Varela et al. in terms of feasibility, reliability and construct validity [11].

The experience of otolaryngology head and neck surgeons in developing assessment tools for evaluating technical skill and knowledge of trainees performing various procedures have been successful as these tools were found to have proven statistical significance [7-9]. This ensures that such validated instruments can be adopted with good educational impact in otolaryngology practice.

This instrument shows promising results to be a reliable, valid and feasible assessment method for testing the surgical skills of the residents within the operation theater, performing thyroidectomy. It provides detailed evaluation with global considerations to the decisive and crucial steps in the surgical procedures that requires competence of the residents. Such instrument can contribute to the educational development in the surgical field as it gives insight and feedback about the competency level of trainees.