This nested-design research study examined classroom factors associated with change in students’ reading attainment and social emotional development during a school year in inclusive grade 3 and 5 classrooms. The study attempted to further investigate components of an effective inclusive teaching model examined by the SET Project [1,2]. Through a nested design, with students nested within classrooms, the present study focused on individual variation in reading attainment and social-emotional development as a function of the quality of classroom teaching, with assessments at two points across a school year in grade 3 and 5 inclusive classrooms.

Inclusive education as an evidence-based practice

Inclusive education is providing appropriate educational practices to students with disabilities by classroom teachers in regular classrooms [3,4]. With a move towards inclusive education [5], classroom teachers are expected to take the leading role in providing differentiated support for all students within their classrooms [3,6]. Evidence from studies of effective teaching reveal that teaching in inclusive classrooms requires not only an understanding of best practices in teaching shared among all school staff but also a knowledge of adapted instruction for students with special needs [6-8]. In addition to having a general and adaptive knowledge of teaching, research findings indicate that classroom teachers’ positive attitudes towards inclusion are amongst the most important factors in creating inclusive classrooms [6,9]. However, reviews suggest that teachers are reluctant to implement inclusive practices [9,10].

Despite a large body of literature on inclusion, there is a lack of empirical evidence on its impact on student outcomes in academic and social-emotional domains [11-16]. A component that is often not included in research examining effectiveness of inclusion and its impact on student outcomes is an observation of classroom teaching. According to Dyson et al. [17], there are limited numbers of classroombased observation studies examining teaching in inclusive classrooms. Farrell [12] suggests that studies should examine observation of actual teaching as opposed to focusing on comparisons of placement labels such as “inclusive” versus “special” classrooms. Thus, focusing on classrooms as units of analysis is a step forward in identifying how inclusive education can be implemented effectively for all students.

Classroom-level factors influencing inclusive practices

Research from the School Effectiveness Research (SER) and the effective teaching literature highlight several characteristics of effective teachers and classroom contexts. Research suggests that effective teachers successfully manage instruction time, have good presentation skills, and regularly monitor learning by questioning and giving immediate feedback [18-20]. Furthermore, classroom climate and teachers’ classroom management skills have been shown to be related to student engagement and achievement [21,22].

In inclusive classrooms, teachers have a key role in creating a classroom atmosphere conducive to learning which may be related to student outcomes. Mackenzie [22] argued that effective teaching is also adapted instruction. Adapted instruction, often referred as differentiated instruction [8], aims to support all learners through a variety of approaches and strategies [18], such as providing choices and multiple types of demonstrations of essential skills [23].

Many argue that differentiated instruction can be challenging and demanding in inclusive classrooms [7,23]. In a study examining teachers’ attitudes towards inclusive education and students’ perceptions of their learning environment, Monsen and Frederickson [24], examined the relationship between teachers’ opinions about inclusive education policies and classroom contexts and found that those students who were in the classrooms of teachers with highly positive attitudes towards inclusive education reported high satisfaction with their classrooms.

Some studies examining teachers’ attitudes reveal that when classroom teachers have resources and support at the school-level, implementing inclusive education is not perceived as stressful [9,25,26]. In a survey study, Bunch et al. [26] examined Canadian teachers’ attitudes towards inclusive education and found that teachers were supportive of inclusion as an educational philosophy. However, teachers also stated a lack of professional training and the amount of workload as major barriers in implementing inclusion. Teachers reported professional training, guidance and leadership of school administration and manageable classroom sizes as factors that can aid the implementation of inclusive practices in classrooms.

A series of Canadian studies titled as the Supporting Effective Teaching (SET) research project found that teachers’ and school principals’ attitudes towards inclusion predicts the presence of observed effective teaching practices in inclusive classrooms [1,2]. However, these studies while important, did not examine whether teachers’ practices or attitudes towards inclusion have any impact on students’ growth, either in academic or social-emotional domains.

In summary, there is limited research examining the relationship between teachers’ attitudes, beliefs, their teaching practices and student outcomes in inclusive education literature. Importantly, the field lacks evidence obtained from classroom observations of inclusive practice [17]. Thus, the present study aims to fill this gap in inclusive education literature by examining the relationship between teachers’ attitudes towards inclusion, their effective classroom teaching practices and the impact of this relationship on growth in student outcomes for both attainment and socio-emotional development.

Research questions

The following research questions were addressed in this study (Figure 1):

Figure 1: Attitudes, practices and outcomes model of the present study.

1. What is the relationship between teachers’ general teaching practices, their adapted teaching practices and their perceptions of inclusion in inclusive Grade 3 and 5 classrooms? (Link A in Figure 1)

2. What classroom-level factors are significantly related to student outcomes of reading attainment, social inclusion, self-concept, bullying and perceptions of inclusion in inclusive grade 3 and 5 classrooms? (Link B and C in Figure 1)

Participants

This classroom-based research study involved 180 grade 3 (n=72) and grade 5 (n=108) students and their 15 classroom teachers in two English speaking school boards located in the Greater Region of Montreal, Quebec, Canada. All participating schools were located in small and rural towns off the Island of Montreal. 44 elementary school principals from two English-speaking school boards were initially approached. Of those 44 principals, eight principals agreed to participate in the study. Once teachers accepted to participate in this study, informed consent forms were distributed to all Grade 3 and 5 students. The final sample consisted of 180 student participants selected from 15 Grade 3 and Grade 5 classes of eight elementary schools from two English-speaking school boards.

Classroom-level sample: In Canada, there is great variation in the terminology, implementation and practices of inclusive education across provinces [27,28]. Currently, all English speaking school boards in Quebec have formally adopted an inclusive philosophy of education endorsed by the Success for All Policy (MELS, 1999). In this sample, classroom teachers were presented with a background questionnaire that was part of the Teacher’s Perceptions of Inclusion (TPI) study [6,29]. Of the participating 15 teachers, 12 were female, and 3 male, 53.3% had spent more than 10 years of classroom teaching. 60% of the teachers reported that they received some training in special needs education. Using these broad categorizations from the TPI questionnaire, 20% of the teachers said that their knowledge of the local legislation or policy as it pertains to students with special needs/disabilities was ‘poor’. Similarly, when asked about their level of experience teaching a student with special needs/disabilities, 26.7% reported having ‘high’ (at least 30 full days) and 46.7% having ‘very high’ (more than 40 full days) levels of experiences. Similar to their experience levels, 73.3% of the teachers reported having ‘average’ and’ above average’ levels of confidence in their teaching of students with special needs/disabilities.

Student-level sample: The total student sample was 180 grade 3 and 5 students (50% female, 50% male) (age M=10.02 years, SD=1.06 years). At the beginning of the research study, parents received a background questionnaire and the informed consent forms. The purpose of this questionnaire was to understand and describe the sample characteristics including the language(s) spoken at home, parental language(s), parental education level, home literacy environment, and students’ official documentations of learning or developmental disabilities (sensory-motor, intellectual, emotional). A chi-square test was used to compare the obtained data of home language of the sample to the expected data based on the population statistics provided by the Statistics Canada’s 2006 census data (http://www.statscan.gc.ca/) to ascertain if the sample is representative of the general population of families in Quebec. Sample data differed significantly from the expected distribution of languages spoken at home among families in Quebec, χ² (3,180) =1831.474, p<0.05. In this sample, there was a greater percentage of English speaking families relative to the Quebec population.

The sample parent self-report data differed somewhat from the expected distribution of education levels within the Quebec population, χ² (4,360) =142.271, p<0.05, in that this sample had more fathers and mothers who had attended tertiary education institutions and fewer who graduated with a Bachelor’s or graduate degrees compared to regional education level population statistics reported by Statistics Canada (2006).

Measures

This study aimed to use well-established measures with good reported reliability and validity.

Standardized measurement of reading attainment: Reading attainment was assessed using a standardized test called the Group Reading Assessment and Diagnostic Evaluation [30]. The GRADE is a norm-referenced group reading test with Canadian validation. The purpose of the test is to identify reading skills that students already possess and skills that may need some further development, and to monitor the growth of students’ reading competencies [30]. The GRADE includes subtests for measuring a range of essential and complementary reading skills such as listening comprehension, word reading, vocabulary, sentence comprehension and passage comprehension. According to a review by Fugate [31], the GRADE is a well-designed tool to measure reading growth as students move up to older grades. The GRADE is research-based and norm-referenced [30]. Concurrent and predictive validity was assessed using a variety of other standardized reading assessments (e.g. TerraNova, Iowa Test of Basic Skills, California Achievement Test, etc.) and correlates highly with these standardized tests [30]. All of the reliability coefficients for alternate form and test-retest are in the 0.90 range indicating strong reliability estimates for the total test at all grade levels [31]. The GRADE is reported to have strong internal consistency (r=0.95-0.99) and retest reliability (r=0.80) [30].

Curriculum-based measurement (CBM) of reading attainment: The Dynamic Indicators of Basic Early Literacy Skills [32] was used to assess students’ reading skills and to monitor their reading competency over the academic year. The DIBELS is a research-based, CBM tool that is freely-accessible on the Internet (http://dibels.uoregon.edu/). Studies reveal correlations of 0.70 with the Florida Comprehensive Assessment Test and 0.73 with the North Carolina end-of-grade assessment tests [33].

Standardized measurement of students’ self-concept: The Tennessee Self-Concept Scale-2 (TSCS 2) which is one of the most commonly utilized tools for measuring multidimensional self-concept for children and adults [34] was administered to measure students’ self-concepts. The TSCS 2 Child Form consists of 76 self-descriptive items that describe how an individual portrays what s/he does, likes, and feels and can be used with ages 7 to 14. A shorter version with only the first 20 items of the original 76 items can be administered to obtain a summary of overall self-concept [34]. Items in the TSCS 2 are self-descriptive statements that the students rate as “Always False”, “Mostly False”, “Partly False and Partly True”, “Mostly True”, or “Always True”. Responses are weighed from 1-5 points. Higher scores indicate higher self-concept measures. The TSCS 2 has high internal consistency (r=0.91) and reliability scores (r=0.72) for total selfconcept. Validity has been established through comparison with other similar instruments such as the Piers-Harris Children’s Self-Concept Scale [35] (r=0.71).

Sociometrics for determining social inclusion and friendships: The Social Inclusion Survey (SIS) [36] is a whole class survey that determines social interaction patterns, friendships and degree of affiliations between students. The SIS is a sociometric technique which requires students to indicate whether they would like to “Like to Work” (LITOW) and “Like to Play” (LITOP) with each member of their class through two questionnaires [37]. Frederickson and Furnham [36] have reported that over a 5-week period the test-retest reliabilities for acceptance and rejection have been at 0.70 to 0.78.

Surveys for measuring frequency of bullying victimization: My Life in School Checklist (MLSC) [38] is a student survey administered as a whole class to determine students’ self-reports of bullying. In a report developed by the Department for Education in England, the MLSC is reported as an example of a “standard questionnaire” along with the Olweus Questionnaire in measuring types and levels of bullying at elementary and secondary schools [39]. Additionally, the US National Center for Injury Prevention and Control present MLSC as one of eight assessment tools to identify bullying victimization [40]. The MLSC is a 39 item survey which includes statements describing positive events and negative events. Items describe different things that might happen to a student in school during any one week. Students indicate whether they experienced the situation indicated in each statement by checking “not at all”, “only once” or “more than once”. The MLSC has been found to have strong split-half reliability and show a high level of correlation with the Olweus Questionnaire [41]. According to Hamburger et al. [40], split-half reliabilities for two samples are 0.80 and 0.93.

Classroom observations for measuring effective teaching: Effective teaching was determined by classroom observations of language arts and literacy instruction using the Classroom Observation Scale [7,20,29]. The theoretical background of the development of the COS is based on the effective teaching ratings utilized by Brophy and Good [42]. Stanovich and Jordan [20] report a 78% mean percentage agreement between two independently coding observers across 33 teachers. McGhie-Richmond et al. [29] state that the inter-rater reliability between two raters is 94% across 63 teachers. The COS contains 27 items grouped in four categories: classroom management, time management, lesson presentation and adapted instruction. The items in each section require the observer to code whether the teacher exhibits each practice “consistently” (twice or more), “inconsistently” (once), “not in evidence” or “not applicable” [2]. Each item is rated on a three-point scale with items receiving 0 for “not in evidence”, 1 for “inconsistently” and 2 for “consistently”. In addition to the COS total score, the observation tool produces four additional scales: 1) Predominant Teaching Style (PTS), 2) student engagement (SE), 2) adapted instruction with a student with special needs (TID), 3) adapted instruction with a student with special needs and 4) adapted instruction with a student at risk for academic difficulties (TIAR) [2]. The relationship between effective teaching (measured by the COS Total score), and student-teacher interaction (measured specifically by the PTS, TID and TIAR scales of the COS) has been examined in several studies within the SET research project [2].

Surveys for determining perceptions of inclusion: Teachers’ attitudes towards inclusion were examined by a Canadian tool called the Teacher Perceptions of Inclusion Survey (TPI) [29], adapted from the Diversity, Individual Development, Differentiation Survey (DIDDs; [27]). There are five versions of the same survey to be used with classroom teachers, students, school administrators, school professionals including special needs educators and parents. Similar to the DIDDs, the new survey is comprised of a 5-point Likert scale where 1=Strongly disagree, 2=Disagree, 3=Neither agree nor disagree, 4=Agree, and 5=Strongly agree. The items on the scale are ordered according to a random number chart, and include a number of “reverse coded” items. A high mean score on an item, factor, or the full scale indicates a positive response (taking into account reverse coding).

Procedure

Data collection continued throughout the entire school year in three distinct phases: student outcome measures twice at the beginning (September-October) and end (April-June) of school year, effective teaching and attitudes towards inclusion measures once in the middle (December-March) of the school year. Students were assessed twice both in Fall and Spring Terms to measure change and value-added gains in their reading attainment and social-emotional growth. The value-added approach looks at the gains of students from one measurement to another and takes into consideration how much students have gained instead of where they stand at a standardized test administered at a given time in a school year [43]. In Winter Term, classroom observations were conducted in language arts and literacy lessons at the convenience of participating teachers. A pair of research assistants observed every classroom using the COS. Each teacher was observed during language arts and literacy instruction for a minimum of two and a half to three hours. Likewise, the perceptions of inclusion surveys were administered once in the middle of the school year.

Descriptive analyses

Missing date reflected less than 1% of the total, and were found to be missing completely at random according to Little’s MCAR test, χ² (57) =49.02, p=0.765, so were imputed using standard procedures in SPSS. Following screening, the appropriate standard data transformation procedures described by Tabachnik and Fidell [44], square root transformations were undertaken on Physical Bullying at pretest (s=1.981, k=2.938), Physical Bullying at posttest (s=2.620, k=6.196) and Verbal Bullying at posttest (s=2.517, k=5.844), normalizing the distribution of the variables. All further analyses with both transformed and original data on these variables were run and it was found that results remained the same at all times. Thus, analyses of the original raw data of the variables are reported in the following data analyses. Descriptive statistics for student-level attainment and self-report data are presented in Table 1 and classroom-level teacher observation and self-report data are reported in Table 2.

Table 1: Descriptive statistics for attainment and socio-emotional measures.

Table 2: Descriptive statistics for teacher observation and self-report variables.

Results for research question 1: What is the relationship between teachers’ general teaching practices, their adapted teaching practices and their perceptions of inclusion in inclusive Grade 3 and 5 classrooms?

In order to evaluate the relationship between teachers’ teaching practices and perceptions of inclusion, two-tailed Pearson product moment partial correlations controlling for grade level were calculated. Teachers’ Perceptions of Inclusion (TPI) score and their total COS score indicating their general teaching effectiveness were significantly associated (r=0.535; n=15, p=0.048), suggesting that teachers’ perceptions of inclusive education are strongly related to their general teaching practices. Two adaptive teaching scales, Teachers’ Interaction with a Student with a Disability score (TID) and Teachers’ Interaction with a Student At risk for Academic Failure due to Reading Difficulties (TIAR) did not show any significant associations with the overall effective teaching score or teachers’ perceptions of inclusion.

Results for research question 2: What teacher-related factors are significantly related to classroom-level variance in growth for student outcomes of reading attainment, social inclusion, self-concept and bullying in inclusive grade 3 and 5 classrooms?

This research question examined the impact of contextual influences in classrooms on students’ variation in reading and socialemotional development. Hierarchal Linear Modelling (HLM) analyses were used for this question given that HLM has the strength to take into account variation both at the classroom- and student-levels [45,46]. This question was answered using a two-level hierarchical model that enabled modeling of main effects of student-level and classroom-level variance.

Data in educational research often have a nested structure, a structure that is ignored in Ordinary Least Squares (OLS) regression. Multilevel models are an extension of OLS regression and follow all assumptions of OLS regression as well as taking into account the shared variance amongst individuals nested in similar groups [45,47]. According to Hox [45] with OLS regression, nested data are moved from one level to another either by aggregating or disaggregating scores, with loss of information and power in further statistical analysis in either option. HLM, on the other hand, has the advantage of retaining information at both levels of data, the individual and the classroom, and is therefore a much more powerful tool when working with nested data [47]. HLM analyses allow for studying relationships at any level in a single analysis by considering the variability associated with each level of the hierarchy [45,47,48]. Thus, the HLM approach to data analysis is an appropriate fit for the second research question of this study, as the aim is to measure student outcome changes over time observed in various students nested in different classrooms. HLM assumes that students from the same classroom share certain characteristics [45,47].

In this study, there are 2 levels of analyses: Level 1 is “students” and Level 2 is “classrooms”. Level 1 and 2 data were combined such that the Level 2 values were identical for all cases representing Level 1 data [49]. As the study had a pre-test to post-test design, an ANCOVA model was appropriate in this design, with pre-test measures as the covariates in each case. This allowed growth in attainment to be examined. This model was fit to determine the effects of the Level 2 predictor variables on different student outcome variables, after controlling for initial skill level using the pre-test covariate. Three SYNTAX commands were used to build the final model that examined main effects both at the student- (Level 1) and classroom- (Level 2) levels with SPSS MIXED Random Intercepts and Slopes model [50]. First, analyses were undertaken to examine whether there is any classroom-level variance with the unconditional model, which is essentially identical to a oneway ANOVA with random effects [49]. Next, the analyses focused on whether there is any classroom-level variance after controlling for pretest and grade level covariance. In the final model, controlled for the clustering of students within classrooms with students (Level 1) nested within classrooms (Level 2), the COS is a fixed effect whereas reading at pre-test in classrooms is a covariate at both classroom (covariate)- and student-levels, (as an unfixed random effect at Level 1). The intercept is also ‘fixed’ and the residuals are modeled as random effects in all models.

These hierarchal analyses of six student-level measures of reading attainment and five student-level measures of social-emotional outcome variables across n=15 classrooms and n=180 students. Overall, results show that there is a significant effect of effective teaching practices measured by the Total COS score for Vocabulary F (8, 139.058) =2.079, p=0.042, Listening Comprehension F (8, 40.602) =2.518, p=0.025, and Reading Fluency F (8, 37.75) =3.602, p=0.003. Social Inclusion variables of LITOW F (8, 80.98) =3.40, p=0.002 and LITOP F (8, 85.22) =9.395, p=0.000 also showed significant effects. Teachers’ Perceptions of Inclusion showed to be significantly related to four out of six reading attainment measures; Vocabulary F(8, 134.548) =9.395, p=0.000, Comprehension F(8, 129.052) =2.511, p=0.014, Total Reading Score F (8, 133.503) =3.883, p=0.000 and Reading Fluency F (8, 43.477) =3.329, p=0.005. Furthermore, three out of five social-emotional variables were found to be significantly related to teachers’ perceptions of inclusion. These were the social inclusion variables of Likes to Work F (8, 79.215) =2.691, p=0.011) and Likes to Play F (8, 77.804) =6.601, p=0.000) as well as Verbal Bullying variable F (8, 63.759) =2.539, p=0.018).

In sum, all classroom-level factors including various teaching practices and perceptions of inclusion were found to show significant impact on student outcomes particularly for Reading Fluency and Social Inclusion. Results showed no overall significant effects of teachers’ practices or perceptions of inclusion on the measures for Teaching Style for Students at Risk for School Failure due to Reading Difficulties (Table 3).

Table 3: RETELL take out words ‘F values (P)’.

The first research question aimed to examine the link between teachers’ general teaching practices and their perceptions of inclusion. It was predicted that teachers who are effective in their teaching also have positive perceptions of inclusion. Correlational analyses showed that there is a relationship between teachers’ effective teaching practices and their perceptions of inclusion when controlled for grade level. Teachers with higher scores on the classroom observation tool for effective teaching also had more positive attitudes towards inclusive education as observed by the teachers’ perceptions of inclusion survey. Similar to the findings of the SET studies [1,2,6,20], results of the present study shows that when teachers have positive perceptions of inclusive education, they also teach more effectively for all students.

After establishing the association between teachers’ attitudes of inclusion and their effective teaching practices, the second research question aimed to investigate if any classroom-level factors of teachers’ teaching practices and attitudes are significantly related to student outcomes in inclusive grade 3 and 5 classrooms. HLM analyses showed that particularly teachers’ general effective teaching practices and their perceptions of inclusion have significant impact on several components of both students’ reading attainment and social-emotional development. Specifically, main effects were found for the overall teaching effectiveness: The Total COS score predicted three reading attainment outcomes of vocabulary, listening comprehension, reading fluency and two social-emotional variables of students’ nominations of peers for both play and work contexts. Likewise, attitudes towards inclusion predicted seven out of a total of eleven student outcome variables including four reading attainment (vocabulary, comprehension, total reading score, reading fluency) and three socialemotional outcomes (social inclusion for both play and work contexts, verbal bullying). In classrooms of teachers with positive views of inclusion, all students’ showed improvement in reading and they also nominated more peers to play/work with.

There are several limitations in this study. First and foremost, this was a study that examined only relationships and associations between variables. Thus, causal links should not be derived from any of the findings. Additionally, the present study is limited given its’ modest sample size at the classroom level (n = 15 classrooms). Thus, this sample size may have resulted in limited power to detect differences using HLM. In addition to the power of sample in hierarchical analyses, there were issues regarding the representativeness of the population characteristics. In the sample, there were more English speaking students compared to the general Quebec population given that the sample was drawn from English-speaking school boards. There were also more parents of participating students graduating from tertiary education colleges as opposed to higher education degrees compared to the regional population statistics. Finally, conducting more rigorous training of observers and adding in a pilot project phase for classroom observations to improve coding and to determine a satisfactory level of inter-rater reliability prior to data collection, would be highly recommended to ensure high quality during classroom observations.

Evaluation of inclusive education practices should focus on environments surrounding the students such as classroom and school contexts as opposed to focusing merely on the deficits of individual students [17,51,52]. The present study examined value-added gains to measure changes in students’ reading attainment, self-concept, bullying and social inclusion in order to observe change in student growth. A contribution of the present study is that it looks at natural variation for the first time using hierarchical analyses in the inclusive education field. Furthermore, this study is also a first attempt to look at growth in both attainment and social-emotional domains at classroom-level. The findings of this study suggest that teachers’ perceptions of inclusion may have instructional implications that may have a relationship to student outcomes. Understanding the interrelationships between teachers’ attitudes, their effective teaching practices and impact on students’ development can inform higher education institutions in providing appropriate teacher education in inclusive education.

This study, a nested design, with students nested within classrooms, was utilized to explore individual variation in reading attainment and social emotional development as a function of classrooms, with assessment at two points across a school year in grade 3 and 5 inclusive classrooms to assess growth in learning. The present study demonstrated, using well-established standardized measures that in inclusive classrooms, student outcomes are associated with teachers’ effective teaching practices and their perceptions of inclusion. For many students, school is the only environment that they can receive interventions and instruction designed to meet their specific needs. Thus, the model discussed in this study, highlighting the relationship between attitudes-teaching practices-student outcomes can be seen as an analysis of a microsystem, one albeit crucial piece of a broader developmental eco-systemic model. Viewed as such it has the potential to lead to an integrated approach to working within complex educational contexts and influences on development. We argue that understanding such links within this theoretical framework will guide in finding pathways to successful implementation of inclusion. As a contribution to the field of inclusive education, this study provides a modest, but we think, important empirical step towards the shared goal of understanding and supporting both the attainment and social-emotional development of all students in today’s inclusive classroom contexts.