Health professionals are being increasingly called upon to effectively integrate behavioral health practices with families and pediatric patients in primary care settings. Motivational interviewing (MI) is a collaborative conversation style for strengthening a patient’s motivation and commitment to change [1]. MI has a strong evidence base across a range of behavioral domains [2] and has been successfully applied in medical settings [3] including pediatrics [4]. Successful applications of MI with pediatric patients include obesity prevention [5] promotion of breastfeeding [6] and reduction of substance use [7] Despite the strong evidence base for using MI with families and pediatric patients, little is known about effective continuing education strategies, particularly for non-trainee health professionals who are already in practice.

Various MI continuing education training formats have been shown to impact providers’ knowledge, self-efficacy, interest in MI, and willingness to use MI [3-7]. However, few studies have used validated measures to observe provider-patient interactions before and after training to assess changes in MI skillfulness [3]. MI is a complex skill that takes considerable time to learn and master [8]. Studies of counseling professionals in non-medical settings have shown that gains in proficiency from workshop-based learning are not sustained unless there are opportunities for follow-up performance feedback or coaching [9]. Despite these findings, few reports of trainings in medical settings incorporate this learning strategy [10].

Of the available studies targeting health professionals working with pediatric populations that included objective measures of MI use, Bohman [11] trained nurses in child health services with a 3.5- day workshop, one objective feedback session, and four individual supervision sessions based on practice samples. The authors were surprised to find no gains in proficiency based on Motivational Interviewing Treatment Integrity Code (MITI) indicators [12]. In addition, few providers met beginning proficiency standards following training and supervision. Another study, which targeted pediatric residents, found more promising results following two 4.5- hour workshops and one performance feedback session, with small improvements in a composite measure of MITI-rated behaviors (open questions, reflection, and MI-adherent behaviors) noted during Objective Standardized Clinical Evaluations (OSCE) at seven months.

The current study sought to clarify the mixed evidence regarding the effectiveness of MI training with pediatric health professionals, by investigating the impact of a four-hour workshop plus performance feedback and coaching on the ability of pediatric health professionals to employ MI techniques with a standardized patient.

Participants

Study participants were health care professionals who worked with pediatric patients in a large, urban, safety-net hospital, affiliated with a medical school. Participants were recruited to participate in the training as part of a day-long retreat.

Overview of procedures

Potential participants were invited to participate in the research evaluation via email. The e-mail message described the study and linked participants to an electronic consent document. Informed consent included information regarding their right to refuse participation and to withdraw at any time, without affecting their ability to participate in the retreat. This study was approved by the University of California San Francisco Committee on Human Research. Consenting participants were forwarded to an online scheduler where they selected a time to complete the baseline assessment (BL) via telephone. BL included basic demographic questions and a recorded standardized patient interview that served as the baseline practice sample. Practice samples were rated with the MITI 3.1.1. [12]. Participants received written standardized performance feedback prior to participating in the four-hour MI workshop training. Participants submitted another practice sample one-week following the workshop (FU1), and were then contacted via email or telephone by an MI coach to schedule a 30-minute telephone coaching call. Coaching calls occurred within 4-weeks of the practice sample and were supplemented with standardized performance feedback that was emailed to the health professional. Participant’s submitted one additional practice sample two months following the workshop (FU2). Participants were provided with a $20 gift card for the completion of each of the three assessments.

Practice samples

Practice samples were gathered using a standardized patient (SP) telephone interview format at BL, FU1, and FU2. The SP interview presented participants with one of three case presentations that were randomly selected and balanced across assessment points. Case presentations included: 1) a risky drinking adolescent, 2) an obese adolescent, and 3) a postnatal mother contemplating discontinuation of breastfeeding. Providers were read brief background information about the patient and were then instructed to take up to ten minutes to counsel the patient about the targeted behavior. The SP was trained using standard procedures [13], including a general orientation to the cases, read-throughs, question/answer opportunities, role-playing, and feedback.

Coding of practice samples

Interactions were digitally audio recorded and coded using the Motivational Interviewing Treatment Integrity (MITI) Code 3.1.1. [12]. This instrument is the most widely used MI fidelity instrument, has demonstrated reliability and sensitivity [14], and is predictive of outcome for a range of behaviors [15]. The MITI has two components: global scores and behavior counts. Global scores are intended to capture holistic aspects of the intervention. MI spirit is a global score that combines the MI constructs of evocation, collaboration, and autonomy / support. It is assessed on a five point Likert-scale (0=low, 5=high) that is anchored in corresponding qualitative descriptions of increasing competence in each area. Behavior counts are sums of specific provider utterances such as reflections (comments back to the patient that capture the essence of what was just said), open-ended questions, and MI adherent behaviors (affirmations, asking permission). Beginning proficiency and competency thresholds are provided for MI spirit, reflection to question ratio, percent open questions, percent complex reflections, and percent MI-adherent behaviors. The MITI coders included two authors (J.H and J.M.), both of whom have extensive rating experience. The MITI specific beginning proficiency and competency standards can be found in Table 1.

Table 1: Practice sample performance across time.

Motivational interviewing training

All participants completed a four-hour motivational interviewing (MI) training as part of the retreat. The training was led by five of the authors (A.W, J.H, J.M, C.C, M.T). At the time of the workshop, four of these trainers (J.H, J.M, M.T, C.C.) were members of the Motivational Interviewing Network of Trainers (MINT). Participants were provided with a pocket card with prompts for conducting an MI-based brief behavioral counseling intervention. The training was anchored on this protocol and focused on understanding MI spirit, practicing MI core interviewing skills (open-ended questions, affirmations, reflections, and summaries), facilitating exercises designed to build motivation (scaling rulers, values clarification, and decisional balance), and behavior change planning.

Feedback and coaching

Written MITI-based feedback based on BL was provided to participants prior to the MI training. This included a feedback sheet that listed global scores, behavior counts, and summary scores for the provider. Beginning proficiency and competency standards, as well as explanation of the meaning of global scores, were provided for reference. Several sentences of qualitative feedback regarding strengths and areas for improvement were also provided. Written feedback and coaching were also provided based on FU1. Coaching calls were conducted via telephone by the MI trainers. During the coaching call, the coaches used principles of MI to discuss the providers’ scores, explored their perceived barriers to using MI, discussed ways in which the provider could change their behavior to be more MI-consistent, and facilitated role-play practice of specific skills.

Analyses

Paired samples t-tests were used to investigate the significance of group changes from baseline to each follow-up point and from FU1 to FU2 for each of the measured MITI constructs. In addition, because of concerns regarding low power from small sample size, unbiased estimators of effect size, d, and 95% confidence intervals were calculated. Comprehensive Meta-Analysis (Version 2) software was used for all meta-analytic v16] Cohen’s [17] criteria for identifying the magnitude of an effect size were used, where d = 20 is a small effect, d = 50 is a medium effect, and d = 80 is a large effect.

Of 22 health care professionals invited, 11 (50%) consented to participate in the study and all completed the baseline assessment. The participants consisted of physicians (N=7), nurse practitioners (N=2), and developmental clinicians with master’s degrees (N=2). No demographic data is available regarding non-participants that can be used to compare those who chose to participate versus those who did not. All were female and the average age was 35. Regarding baseline confidence in their MI skills, 7/11 participants choose a number < 5 on a scale from 0 (low) to 10 (high). All subjects were retained at FU1, 8/11 (73%) completed a coaching call, and 7/11 (64%) completed FU2. Independent samples t-tests revealed that there were no significant differences between those who completed and those who did not complete FU2 on age or any MITI measures at baseline of FU1.

MI fidelity ratings at each assessment period are shown in Table 1. At the baseline assessment, fidelity to MI was highly variable. MI Spirit ranged from 1.66-4.66 (M=3.39, SD=95), with 64% of providers meeting beginning proficiency and 36% achieving competency standards. In contrast, beginning proficiency use of open-ended questions, complex reflections, and MI adherent behavior was low (27%, 18%, and 37% respectively), and no provider demonstrated beginning proficiency in reflection-to-question ratio.

From baseline to FU1, all MITI performance measures improved. We measured statistically significant increases in mean reflection to question ratio (13%, 33%), open-ended questions (37%, 50%), and complex reflections (13%, 58%; Table 1). At FU1, 64% of participants achieved both the baseline proficiency and competency standards for MI adherent behavior, although the increase in mean MI adherence from BL to FU1 (71%, 92%) was not statistically significant. No participants met beginning proficiency for reflection to question ratio.

From FU1 to FU2, mean performance measures of MI Spirit, reflection-to-question ratio, open-ended questions, complex reflections, and MI adherent behavior all decreased, but these decreases did not reach statistical significance. Effect sizes ranged from small to large (Table 1). The proportion of participants meeting beginning proficiency or competency standards decreased between FU1 and FU2.

From BL to FU2, there were no statistically significant changes in MITI behavior counts, but effect sizes ranged from small to large (Table 1). Again, there was no increase in proportion of participants meeting beginning proficiency or competency standards between BL and FU2.

In the current study, we found that a four-hour MI continuing education workshop plus feedback and coaching for pediatric health professionals produced improvements in objective measures of MI performance with medium to large effect sizes in every measure. Interestingly, however, the percent of participants reaching competency levels were quite low.

This is perhaps best exemplified by reflection to question ratio, which improved significantly from BL to FU1, despite the fact that no participants approached beginning proficiency standards at any time point. Additionally, despite receipt of feedback and coaching between the workshop and FU2, decreases in performance were observed across this two-month time span. Perhaps the coaching could have been more effective had it happened more immediately after the provider participated in the mock interview; some coaching occurred up to one month later due to logistical constraints. Results of our study suggest that, although a continuing education half-day workshop and feedback with coaching improves pediatric providers’ MI skillfulness as measured by a well-validated evaluation tool, these gains may not be large enough to be clinically meaningful, and either more intensive or frequent training may be needed. Given the difficulties our health professionals had in particular with reflective listening, future trainings might be more effective if they focus more intensely on this behavior.

Our results also suggest that it is feasible to gather practice samples and administer feedback and coaching via telephone with health professionals who work with families and pediatric popoulations, which, to our knowledge, has not been previously reported in the literature. Feedback and coaching over a longer period of time, with greater frequency, and with actual patient encounters may produce greater improvements. Given the logistical challenges for busy clinicians to schedule in-person feedback and coaching sessions, however, the innovative use of digitally recorded practice samples, asynchronous on-line scheduling at the convenience of the learner, and telephone-based assessment and coaching may improve MI training and implementation efforts in medical settings.

The current study has several important limitations. The sample size was small and did not include a control group, limiting generalizability and our ability to make inferences about causality. Prior exposure to MI was also likely variable in our sample, given that several participants rated their confidence as on the upper end of a scale even prior to the training.

MI is a complex skill that, when practiced well, has the potential to help patients transform unhealthy behavior. Feedback and coaching may help boost MI skillfulness after a training workshop, but the minimum effective amount and type of this feedback and coaching has yet to be established. While the tested training model led to improvement in skillfulness, and telephone feedback was feasible, additional training is likely needed to meaningfully impact behavior.

The authors wish to thank Eileen McCormick for participating as the standardized patient. Funding-This work was supported by SAMHSA grant U79 TI020296 (Lum) and K23 AA020865 (Hettema).

Disclosures: Dr. Jennifer Hettema serves as a steering committee member for the Fetal Alcohol Syndrome Disorders Regional Training Center at University of Nevada Reno. She is also an MI training consultant for Health trust, a nonprofit foundation. As a consultant, she trains a group of master trainers so that they may teach others how to conduct MI. She also discloses that she owns two limited liability corporations that sell motivational interviewing (MI) training videos and curricula that she developed.