*Keywords:* Reading comprehension, Assessment, Neale Analysis of Reading Ability, Maltese, English

‘louisa-grech’, ‘paul-bartolo’, ‘liberato-camilleri’, ‘john-everatt’

---

Volume 12, No.2. 172 195 Faculty of Education©, UM, 2018

English Reading comprehension assessment in Malta:

An evaluation of the use of the Neale Analysis of Reading

Ability

Louisa Grech

University of Malta

Louisa.grech@um.edu.mt

Paul Bartolo

University of Malta

paul.a.bartolo@um.edu.mt

Liberato Camilleri

University of Malta

liberato.camilleri@um.edu.mt

John Everatt

University of Canterbury, New Zealand

john.everatt@canterbury.ac.nz

Abstract: English literacy assessments in Malta are mainly based on tests standardised among English speaking populations. Such practice raises questions as to its suitability within the Maltese bilingual context. This study explores the implications of such practice by evaluating the performance of Maltese students on a widely used test, namely the Neale Analysis of Reading Comprehension – Revised (NARA II). The test was individually administered to a total of 224 students, representing developmental processes through three age-groups between 7-14 years, and two different home language backgrounds also reflected in two different types of school, State and Independent. Analysis of variance and correlations were used to highlight differences between the scores of the three age-groups and the two types of school and home background. The results indicate that comprehension levels improved sequentially with age; however, the norms derived from a monolingual English population do not represent appropriately the varying bilingual contexts of Maltese students. Moreover, particular discrepancies in the participants’ scores on the three subscales of the test

• namely Reading Comprehension, Reading Accuracy and Reading Rate raise questions about the profile of competencies in English reading comprehension that are being acquired by Maltese students.

Keywords: Reading comprehension, Assessment, Neale Analysis of Reading Ability, Maltese, English

Introduction

This study attempts to address the need for more accurate assessment of the reading comprehension development and difficulties of Maltese students. Presently, a number of foreign standardised measures are still being used to assess literacy skills and difficulties. In Malta results from these assessment measures are an important deciding factor for: i) the formulation of the statementing of individual educational needs of a child, ii) the provision of additional support in mainstream schools for children with specific learning difficulties, and iii) the opportunity to obtain access arrangements during school and/or national exams (European Agency for Special Needs and Inclusive Education, 2014). However, a number of renowned researchers in the field of bilingual education (e.g. Abedi, Huie-Hofstetter & Lord, 2004; Basterra, Trumbull & Solano, 2011) are questioning the validity and reliability of these tests on students who do not have English as their first language, especially when considering the linguistic and cultural differences between countries.

This study will evaluate the use of the Neale Analysis for Reading Ability – Revised (NARA II) (Neale, 1997) for the evaluation of children’s literacy development in Malta. The NARA is the most widely used measure in Malta to assess English reading skills of word recognition, speed of reading and comprehension, also in assessments for requesting access arrangements in national examinations (Ghirxi, 2013). This study aims at evaluating the adequacy or otherwise of making use of this UK normed test for assessing Maltese bilingual children coming from diverse language backgrounds and different school types in Malta.

Reading Instruction and Assessment in Malta

Despite obtaining independence from Britain in 1964, Malta’s education system remains quite similar to that used in the UK. One of the implications of this influence is that methods of assessment of literacy are still mainly based on tests originally standardised among English speaking populations (e.g. in the UK, the USA and/or Australia). This situation is problematic because Malta is a bilingual country, with most individuals experiencing different combinations of Maltese and English as part of their day-to-day life and schooling. The type of school that students attend also relates to their varying language use. For example, Church and Independent schools are

largely English-speaking, while State schools are more likely to use mainly Maltese or use English-Maltese code-switching in the classroom (Ministry for Education and Employment, 2015). For children attending State schools, the school could ‘constitute the only source of learning in the second language’ (Ministry of Education 1999, p. 23).

Thus, it is questionable whether the content of a UK monolingual Englishlanguage assessment battery would be suitable for children for whom Maltese is the first, or dominant, language (Elbeheri and Everatt, 2016). Although such learners might be capable of communicating in English, their score on a monolingual-based test could be much lower than that of equally abled bilingual learners whose first, or dominant language, is English (Figueroa 1989). The scores of the former might not reflect their true literacy ability, particularly if their scores are compared to the standardisation content and norms for native speakers of English in a different country. The norms of UK standardised English tests might not adequately reflect the profile of children’s development and difficulties in reading comprehension. They might also either lead to over-diagnosis of literacy difficulties or a reconsideration of test’s cut-off points for decisions about children’s learning difficulties. However, no formal criteria currently exist for a lower cut-off point, and the test used might not be sensitive at low-score extremes. Similarly, students might perform well on word level (accuracy) literacy measures but poorly on measures of reading comprehension. Each of these conditions could lead to misdiagnoses of reading skill profiles among the Maltese population of school children. These possible misdiagnoses could lead to inaccurate decision making with regards to individualised school support and/or access arrangement opportunities during examinations.

Procedures for Identification of Reading Comprehension Difficulties

The most practical form of assessing literacy acquisition in young children by psychologists has been through word recognition and spelling skills, which have been part of the widely used British Ability Scales (Elliott, 1983; McBride, 2014). This assessment method has been the practice in Malta based on the understanding that, even though word reading is only a means of getting meaning from print, word recognition skills also reflect the level of reading comprehension (Falzon 1972; Bartolo 1988). The University of Malta (2015) guidelines for examination access arrangements for end of secondary school examinations (SEC and MATSEC) states that “ candidates who are unlikely to be able to read the examination material with sufficient accuracy ” will be

offered arrangements “ to avoid making mistakes which will affect the understanding of what they read ” (p. 40). Here the assumption is that the examiners will only consider arrangements for difficulties in comprehension that may arise from word recognition skills. However, it is increasingly being realised that, even though word-level skills are highly related to reading comprehension skills, and poor decoding skills cause comprehension problems, there is a large proportion of children with specific reading comprehension weaknesses who do not seem to exhibit difficulties at wordlevel reading (see Cutting and Scarborough 2006; Spooner, Gathercole, and Baddeley 2006; Oakhill, Cain, and Elbro 2014). Some 10 to 25 per cent of school-age children exhibit comprehension difficulties despite demonstrating effective decoding skills (Cutting and Scarborough, 2006). This view is further strengthened by a recent study conducted by Grech, Everatt, Bartolo and Camilleri (2017) who found that reading comprehension in Maltese was related not only to word-level processes (e.g. non-word reading), but also to language and grammatical level processes (e.g. listening comprehension and syntactic awareness). Moreover, while word level processes were more predictive in the primary years, linguistic and grammatical factors were more predictive in the secondary levels.

Literacy Assessment Tools in Malta

Falzon (1972) and Bartolo (1988) developed Maltese single-word reading tests for primary school-age Maltese children. With regards to decoding skills, Bartolo (1988) found that children attending Church and Independent schools scored significantly lower on Maltese word reading tests up to the age of 8 but reached equivalent levels of proficiency in decoding skills in Maltese by the end of primary school (age 10-11). More recently, Agius (2012) conducted a study on the comprehension skills of Maltese students aged 8 and 12 years in both Maltese and English. Her results showed that the language used for assessing was an essential factor in diagnosing reading comprehension difficulties in Malta. She thus concluded that any assessment tools used locally should reflect the bilingual context of Maltese children and the type of school they attend.

Locally developed assessments are therefore needed to determine which Maltese children experience comprehension difficulties, and why, to develop appropriate instruction that meets their individual needs. However, to date, the only locally standardised test (Agius 2012) that includes both single-word reading and text comprehension, and in both Maltese and English, remains

unpublished. Therefore, the only comprehension assessment measures currently available are English sentence reading comprehension tests based on the Suffolk Reading Scale II (Hagley, 2002) with Maltese norms (University of Malta/Access Disability Support Committee & Ministry of Education / Directorate for Quality and Standards in Education, 2010) and the similarly constructed Naqra u Nifhem (Read and Understand) in Maltese. These have a cloze test format which requires the test-taker to fill in missing words in a sentence from a list provided. This format is considered to rely more on single-word reading than comprehension (Cain and Oakhill, 2006). Despite the necessity of testing at sentence level, reading generally involves the ability to make sense of text within a wider context. C urrent evidence (e.g. Oakhill et al, 2014; Cutting et al, 2006) validates the view that reading difficulties can be differentiated between two types of learners: those who have difficulty understanding at word level and those who have difficulty understanding text at sentence and passage level. Understanding requires higher-level language skills (e.g. inference, knowledge and use of text structure, and comprehension monitoring) that go beyond single-word understanding (Oakhill and Cain, 2012).

Reading Comprehension Skills

Reading comprehension cannot be measured along one dimension. Assessment tools need to take into account cultural and knowledge background (Keenan 2016; Peer and Reid 2016, 53), as well as higher (e.g. syntax) and lower (e.g. word level processes) language skills, speed of processing, and storage and memory recall abilities (Oakhill et al., 2014). The evidence so far supports the Simple View of Reading (SVR) (Gough & Tunmer, 1986; Hoover & Gough, 1990) that stipulated that for reading comprehension to occur two basic processes are needed: word decoding and language understanding. Elaborating this idea further, Joshi and Aaron (2000) included speed as an additional component to the SVR model. Speed is now being studied as an independent skill that contributes to reading ability. Being fluent involves the competence to read with appropriate speed, accuracy, and expression (Rasinski, 2006). Unfortunately, many teachers associate fluency solely with the speed of reading, with the assumption that if a student is able to read at suitable speed, then the reader will automatically understand (Marcell, 2011).

As students advance through school, they transition from learning to read, which involves learning to decode and developing fluency and

comprehension skills, to reading to learn, which requires students to use their comprehension skills to learn from the text (Chall 1996). This transition is often most evident in the upper elementary grades (Oakhill et al., 2014) when many readers begin to encounter difficulties with new comprehension requirements (Shanahan and Shanahan, 2012).

It is being increasingly recognised that reading complications might arise also from issues other than comprehension skills, such as a student’s socioeconomic position or if a student is bilingual or experiences learning difficulties (EU High Level Group of Experts on Literacy, 2012 ). Such complications have often led to an under-representation of bilingual students when identifying reading comprehension difficulties because educators more often attribute problems to a language barrier rather than to an alternative reason (Peer and Reid, 2016). The lack of diagnostic materials for bilingual individuals has resulted in uncertainty when distinguishing between performance difficulties stemming from a language barrier and those stemming from a more specific deficit, such as dyslexia – or from both simultaneously (Mortimore, 2012; Everatt, Smythe, Adams, and Ocampo, 2000). This uncertainty is as much an issue for the reliable identification of students who are experiencing literacy learning difficulties as any other area of bilingual assessment. In some cases, this situation has led to a wait-and-see requirement in an assessment procedure, meaning that identification, and hence intervention procedures, are postponed until the individual has gained sufficient competence in English as a second language to be assessed (see discussions in Cline and Shamsi, 2000; Everatt and Reid, 2010).

Assessment for Bilingual Learners

Literature regarding literacy difficulties and bilingualism has often focused on cognitive or linguistic factors and ignored elements of cultural affiliation or diversity and group identification with the language (Cline and Shamsi, 2000). However, simply translating diagnostic measures from one language to another could create more problems than benefits. As put forward by Abedi et al. (2004), students for whom English is not their first language may be at a disadvantage because of lack of familiarity with the complexity of the structures of the second language, and of its linguistic properties and vocabulary. Wainer (2013) among others argued that such procedures result in measures that produce invalid scores, and thus inappropriate assessment conclusions.

In addition, learning to read and write in one language is not necessarily the same as in another, and it may be even more different than bilingual literacy development. An assessment measure might be rendered inappropriate as a predictor of literacy skills by the possibility that underlying cognitive factors related to literacy learning difficulties can vary between orthographies and by the impact of aspects of the language or culture within which an individual is immersed (see discussions in Everatt, Smythe, Adams, and Ocampo, 2000; Goswami, 2000; Ziegler et al., 2010). Reading is a socio-cultural act, where the ability to create cultural models and understanding depends on the environment that the reader is in and the cultural experience to which the reader is exposed (Andersson and Barnitz, 2004). Therefore, a lack of familiarity with culturally friendly texts could result in comprehension errors or misunderstanding. Standardised measures need to have cultural validity ( Solano-Flores and Nelson-Barber, 2001).

Despite the above considerations, few assessment tools are specifically designed to assess bilingual learners. In the vast majority of cases of bilingual assessment, the tools used have been developed for assessing monolingual populations (Valdes and Figueroa, 2004). As noted above, this is very much the situation in Malta. It is important, therefore, in the first place to demonstrate the possible distortions that may be created through this practice. This study is an attempt to take up the issue by exploring the implications of using a UK standardised reading comprehension test with Maltese bilingual students.

Methodology

The aim of the study was to explore the implications of using a UK standardised reading comprehension test with Maltese bilingual children. The test used was the Neale Analysis of Reading Ability (NARA II, Neale, 1997). This test was standardised for a British population aged 6–13 years, but it has been the most widely used English comprehension test in Malta even for adolescents. The NARA has been used both to investigate issues of English reading comprehension development (e.g. Cain and Oakhill, 2006) as well as issues in the diagnosis of reading comprehension difficulties (e.g. Colenbrander, Nickels and Kohnenone, 2017; Wheldall and Arakelian, 2016). The following three more specific research questions were addressed: (1) How do the reading comprehension scores of Maltese students on the NARA II compare to British standardised norms? (2) How do reading

accuracy and reading rate relate to reading comprehension on this test? And (3) How does language background relate to performance in reading comprehension on this test?

Sample

A sample of 224 students aged 7-14 years was intended to represent two main conditions: the performance of students at different ages and levels of proficiency, and the performance of students coming from different language backgrounds. Table I shows how participants were first grouped into three school grade groups, namely Years 3-4 (27%) and Years 5-6 Primary (31%), and Forms 1-3 Secondary (42%). These reflected the three expected developmental stages or phases of reading ability as suggested in the literature (e.g. Chall, 1996; Ehri, 2005; Frith, 1985), namely that between birth and age 9, children learn alphabetic writing that corresponds to the sounds of words, decoding and phonological principles; from age 9 onwards, children’s reading moves steadily to a mature development where (in addition to word recognition skills) increase in vocabulary, syntactic knowledge and abstract thinking are necessary for successful understanding; and the third age group was categorised separately to investigate possible different higher developments in comprehension.

Secondly, the sample consisted of two groups of students coming from different home backgrounds: 54.5% of the sample was from State schools where students come from a mainly Maltese speaking home language background – indeed only 6 of the 122 State school students spoke English at home (see Table II); the other 45.5% were from an Independent school where children came mainly from English speaking home backgrounds – only 21 of the 102 Independent school students had Maltese as their dominant home language background. A small number from both types of school spoke both Maltese and English at home, while a few others had another dominant home language (see Table II). Table I also shows how the sample represented the two genders in order to consider the possibility of gender influence on the reading comprehension scores. However, as no difference was found between the genders, due to the selection of students of average English competency whatever the gender, this variable was not explored further.

All the students selected from the Independent schools and from State schools in the primary levels were from a class of mixed abilities. At middle and secondary State schools (Forms 1, 2 and 3), where students are streamed

according to ability in English as determined by exams, the classes of average ability were selected to avoid possible outliers.

Table I: Number of participants by grade, gender and school type

Grade (Age) Gender

State Schools

Independent Schools TOTALS Yr3–Yr4 (8–9 yrs.)

Male 12 14 62 (27.4%) Female 18 18 Yr5–Yr6 (9–11 yrs.)

Male 22 15 70 (30.6%) Female 21 12 F1–F3 (12–14 yrs.)

Male 31 21 92 (42.0%) Female 18 22 Male 65 50 115 (51.3%) TOTALS Female 57 52 109 (48.7%) ALL 122 (54.5%) 102 (45.5%) 224 (100%)

Table II: Participants’ language background

School type Total State Independent

Language spoken at home

Maltese 95 21 116 English 6 44 50 Maltese and English 17 26 43 Other 4 11 15

Total 122 102 224

Test information and procedures

The NARA II (Neale, 1997) was based on assessment of British children aged 6 to 11.08 years with the norms being extrapolated to 13 years. The test comprises six passages that increase in difficulty and length. There are two equivalent sets of texts and testing forms (Forms A and B). Given the tendency for professionals to make most use of Form A, this study used Form B. The student reads each passage aloud so that reading errors are noted and corrected for the student to proceed with more possibility of understanding, and then the examiner orally asks a set of questions about the text for the student to also answer orally. Each participant was tested individually in a separate quiet room. Participants were first given a demonstration of an example passage at the beginning of the session to ensure they understood the procedures. The test was administered by eight final-year undergraduates in psychology who received 25 hours of training on its administration procedures from the main researcher. The student’s performance yielded three separate scores: (1) a Reading Accuracy score depending on the number of errors made while reading aloud; (2) a Reading Rate score depending on the time taken to read each passage; and (3) a Reading Comprehension score depending on the number of correct answers given to the oral questions for each passage.

The results were analysed using the statistical programme SPSS (24), where descriptive statistics, analysis of variance (ANOVA) and correlation analyses were conducted to answer the research questions.

Results

The data from this study were used to answer three research questions: (1) How do Maltese Reading Comprehension scores on the NARA II compare to British norms? (2) How do their Reading Accuracy and Reading Rate scores relate to Reading Comprehension scores? And: (3) How does language background relate to Reading Comprehension performance? The following analyses were performed to support the development of answers to these questions. Firstly, Table III shows the mean Reading Comprehension age score on the NARA II norms achieved by the three grade groups of participants, compared to their mean chronological age. A one-way analysis of variance on Reading Comprehension age produced a main effect of group (F(2,221) = 41.9, p < .001), and Tukey post-hoc comparisons indicated that comprehension levels

were significantly different (p < .001) for all pairwise comparisons of the three groups. These analyses indicated that as the age of the group increased, so did comprehension score.

Table III: Mean Chronological Age and Reading Comprehension Age score (with standard deviations in brackets) of the three grade groups

N Mean Chronological age^ Mean Comprehension age

Yr3–Yr4 62 7.87

7.87

(1.45)

Yr5–Yr6 70 9.87

9.21

(2.37)

F1–F3 92 12.12

10.61

(1.60)

However, Table III also shows that only the youngest age group of Maltese participants achieved average scores on the British age norms that corresponded to their chronological age. The middle group (chronological age 9.87 years) scored on average approximately at a six-month lower level, while the oldest age group’s average score was well over one year below their chronological age.

An explanation of the discrepancy between the Maltese students’ average scores in comparison to the corresponding NARA II British age-group norms becomes easier when one compares the scores of the students from the State schools to those from the Independent schools, and considers also their scores for Reading Accuracy and Reading Rate (see Table IV).

A two-way ANOVA was performed to assess any interaction effect on Reading Comprehension of age group and school type. This analysis indicated that the two main effects were significant. In other words, the mean Reading Comprehension age differed significantly across the three age groups (F(2,218) = 57.7, p < .001) and across the two school types (F(1,218) = 68.1, p < .001). However, the interaction effect was non-significant (F(2,218) = 0.26, p = .77), suggesting that the effect of grade on Reading Comprehension age does not depend on the school type. This finding implies that the rate of increase in Reading Comprehension age as students proceed from one grade to another is unlikely to be influenced greatly by school type: the steepness of

the lines in Figure 1, representing State and Independent schools, are fairly parallel.

However, the two columns of Table IV on Mean Chronological Age and Reading Comprehension Age show that while, for all three age groups, the State and Independent groups had the same mean chronological age, the State school students’ Reading Comprehension age scores were significantly lower than those of students attending the Independent school ( p < .001). Moreover, while all State school age groups scored below the corresponding British norms, the two younger Independent school groups scored above the corresponding norms for their chronological age, though the oldest Independent group also scored below the corresponding British age norms.

Table IV: Mean Chronological Age, Reading Comprehension, Reading Accuracy and Reading Rate Age scores (with standard deviations in brackets) of the three grade groups from State and Independent schools

Grade School type

Mean Chronological age

Reading Comprehension age

Reading Accuracy age

Reading Rate age

Yr3– Yr4

State 7.68

7.05

(0.92)

7.65

(0.88)

10.40

(1.83)

Independent 7.95

8.63

(1.44)

9.02

(1.52)

10.01

(1.87)

Yr5– Yr6

State 9.86

8.46

(2.49)

9.75

(2.10)

11.49

(1.04)

Independent 9.91

10.39

(1.59)

10.62

(1.44)

11.76

(0.89)

F1–F3

State 12.23

9.71

(1.59)

9.77

(1.68)

11.41

(1.17)

Independent 12.22

11.65

(0.79)

11.97

(0.34)

12.05

(0.31)

Figure 1: Influence of grade and school type on Reading Comprehension scores

The results presented in Table IV also show that Maltese students from both State and Independent schools in the two lower (primary) age-groups scored either at or above chronological age level with regards to Reading Accuracy. Moreover, even though at the older age group, students from both types of school scored below what would be expected based on their chronological age, their Reading Accuracy and Reading Rate scores were still above their Reading Comprehension scores (see Table IV). Students from State schools with a mean chronological age of 12.23 years scored a Reading Accuracy age of 9.77 years, while students attending the Independent school with the same chronological age (12.22 years) scored a mean accuracy age of 11.97 years. A two-way ANOVA was performed to assess any interaction of age group and school type on Reading Accuracy. This analysis indicated that the two main effects were significant: Reading Accuracy age differed significantly across the three age groups (F(2,218) = 56.0, p < .001) and across the two school types (F(1,218) = 54.0, p < .001). In addition, the interaction effect was significant (F(2,218) = 4.06, p = .018), which implies that the rate of increase in Reading Accuracy age as students proceed from one grade group to another is related to the type of school attended: the steepness of the lines, representing State and Independent schools in Figure 2, are not parallel. It appears that for those

in State schools, there is little increase from Years 5-6 to the Forms 1-3 age groups.

Figure 2: Influence of grade and school type on reading accuracy scores

Table IV also presents the results for speed of reading (Reading Rate). All participant groups, except the oldest age group of the State schools, scored on average above the expected corresponding British norms. The youngest (7.96 years) and middle (9.86 years) age-groups of the State schools scored nearly three years ahead of their chronological age, while those from Independent schools scored around two years above the British norms. The same relatively good Reading Rate score occurred in the oldest age groups though the State schools group (12.07 yeas) scored approximately eight months below their mean chronological age, while the same age group (12.15 years) of Independent schools scored at the same level as their chronological age. This may partly be the result of the fact that on the NARA, speed of reading is calculated only on those texts in which the student does not make more than 16 errors. Again, a two-way ANOVA was performed to assess any age group by school type interaction on Reading Rate. This analysis indicated a main effect of age groups (F(2,213) = 30.9, p < .001) but not school types (F(1,213) = .96, p = .328). In addition, the interaction effect was significant (F(2,213) = 3.16, p = .044), consistent with the lines representing the progression of Reading Rate age for State and Independent school students showing contrasting slopes

(see Figure 3) – though the lines also suggest that most of the growth in reading rate occurs within the primary year groups.

Figure 3: Influence of grade and school type on Reading Rate scores.

In order to further assess associations between Reading Accuracy/Rate and Reading Comprehension, partial correlations were performed controlling for school type (State/Independent) and grade group (Yr3–Yr4, Yr5–Yr6, F1–F3). These indicated that the relationship between Reading Comprehension and Reading Accuracy (partial r = .74) was much larger than that between Reading Comprehension and Reading Rate (partial r = .37) – and such differences were found if each grade group were analysed separately. However, if correlations were calculated for the different school types independently, then a different pattern is evident, particularly for the older Independent group cohort (see Table V) for whom the influence of both Reading Accuracy and Rate on Comprehension is much lower.

Table V: Correlations between reading comprehension age and reading accuracy/rate age for each grade group and school type State schools Independent schools

Yr3–Yr4 Yr5–Yr6 F1–F3 Yr3–Yr4 Yr5–Yr6 F1–F3 Reading Comprehension and Reading Accuracy

.65 .79 .72 .80 .81 .28

Reading Comprehension and Reading Rate

.49 .41 .27 .53 .24 .08

As has already been observed, type of school is indicative of differences in students’ overall use of Maltese and English (State school students are more likely to use Maltese, whereas Independent school students are more likely to use English). However, to study more accurately the influence of language background on reading comprehension performance, use was made of the more precise indication of home language background in the students’ self report of language spoken at home. Table VI reports the data on Reading Comprehension for those students who indicated speaking English only or both Maltese and English at home compared to those students speaking only Maltese at home. An analysis of variance on these data indicated a significant effect of language group, and Tukey Post Hoc comparisons showed significant differences between the Maltese home language group and both the English home language and the Maltese and English home language groups (p = or < .01), but not between the latter two groups.

Table VI: Mean Reading Comprehension Age according to language background

N Mean Chronological age^ Mean Comprehension age

Maltese 116 10.42

8.93

(2.10)

English 50 9.92

10.18

(2.07)

Maltese and English 43 10.52^

9.99

(2.13)

Consistent with the findings contrasting State versus Independent school groups, experience of English has a clear influence on the student’s Reading Comprehension performance on the NARA II.

Discussion

What issues arise from using a UK standardised English Reading Comprehension measure? The results show a sequential pattern of progress in reading comprehension levels from the lower to the higher age groups, thus indicating a variable ability from one year group to the next (see Table III). Nevertheless, the norms derived from a monolingual English population generally reflect what is assumed to be the average ability levels of the Maltese population attending Independent schools (and with an English language home background) but do not reflect the average ability levels of those coming from State schools. This finding concurs with the argument posed by Elbeheri and Everatt (2016), who challenge the suitability of such measures for predominantly Maltese speaking students. The present work interprets these findings as indicating that monolingual English-language norms are inappropriate for determining expected Reading Comprehension age levels in a bilingual group, predominantly with Maltese as the first language, such as that comprised by the majority of students in State schools in Malta. The notion that students in Malta attending Independent schools generally score better in English subjects when compared to students from State schools (e.g. results from MATSEC, 2016, and Pirls, 2011) is no novelty. However a deeper view of the results brings about some controversial issues regarding reading comprehension assessment in Malta. Dominant language use amongst the participants leads to the principal concern about whether many students are being wrongly assessed with literacy difficulty, when lower scores on assessments could be caused by language rather than level of reading comprehension proficiency. A similar finding has been reported for students learning English as an additional language also in the UK (Burgoyne, Kelly, Whiteley and Spooner, 2009). This creates problems not only for assessment purposes but also for intervention procedures needed to support the students. These results also concur with those offered by Agius (2012) that literacy testing should be performed in a student’s dominant language.

What do Reading Accuracy and Reading Rate scores tell us?

Though the study showed the inadequacy of using a UK standardised test with Maltese bilingual students, the findings on the discrepancies between the participants scores on the three measures of literacy performance on the NARA II – Reading Comprehension, Accuracy and Rate – are worth considering in more detail. Reading Accuracy and Reading Rate scores for the two younger age-groups were found to be average for children attending State schools and even higher for children attending Independent schools when compared to British norms. Moreover, though the older age-groups’ Accuracy and Rate scores were below their chronological age, they were still above their Reading Comprehension scores (see Table IV). This finding concurs with the findings of Hutchinson, Whiteley, Smith, and Connors (2003) that, during the development of reading, bilingual children tend to score lower on comprehension levels than on accuracy levels and speed of reading, since competences in linguistic ability take longer to be achieved than do word level skills. Moreover, the NARA II like other foreign literacy tests entail vocabulary and cultural knowledge that is important particularly for inferential comprehension questions and would therefore be more difficult to access by Maltese students (see e.g., Burgoyne, Whiteley and Hutchinson, 2013).

The results from this study also found that students in the lower grades performed better in Accuracy than in Reading comprehension. On the other hand, scores for students in Form 1, 2 and 3 were well below average in both accuracy and in reading comprehension. An alarming result was that the scores of State school students indicated that there was no improvement in both Accuracy and Reading Comprehension scores from the middle age group (Years 5-6 primary) to the older age group (Forms 1-3 secondary). These results give rise to the following three major considerations.

Firstly, although a vast majority of research (e.g. Grech et al., 2017; Oakhill et al., 2014) support a strong relation between accuracy and reading comprehension, one needs to consider that percentage (approximately 1025%) of students who do not have difficulty with accuracy but nonetheless perform poorly in reading comprehension. As stated by Oakhill et al. (2014), educational professionals need to differentiate between those students who have difficulty reading single words (and hence comprehension will be poor), and those students who have appropriate word level skills but nonetheless have poor comprehension abilities. The authors concur that accuracy is a

necessary skill for comprehension, but they also think that more weight should be put on reading comprehension beyond word level ability with regards to instruction and assessment procedures. Students as early as age 10 (Oakhill et al., 2014) might be experiencing difficulties that are beyond word level. Yet, if only accuracy levels are being highlighted for assessment identification purposes, then the child with no word level difficulties will undoubtedly fall behind by the time he or she reaches secondary school. These issues are of great importance to the Maltese educational system given the concerns raised by the level of performance of Maltese students on the PISA (2013).

A second worrisome factor is the lack of improvement shown by students in State schools in both reading comprehension and accuracy. As seen from the results, the scores achieved by State school students in Form 1, 2 and 3 were similar to students in Year 5 and 6. One very basic explanation could be that, as students get older, the emphasis changes from learning to read to reading to learn. Thus, teachers at secondary level dedicate less time to word reading, even though words and word knowledge become more sophisticated than in the primary years. According to Zheng (2014), students have less instruction time in the secondary years for language comprehension (e.g. vocabulary learning and word recognition). This finding is of particular concern since influential theories such as the Simple View of Reading (Gough and Tunmer, 1986) have illustrated that decoding and language comprehension are both essential components for reading comprehension to occur. Furthermore there is a need for more research in order to understand which factors are more influential in predicting reading comprehension at secondary levels.

Thirdly, the results obtained for the reading rate of primary school students (see Table VI) indicate that, although some students can read at speeds greater than average, this did not help in achieving a better understanding of text. More specifically results showed that speed was not significant in predicting reading comprehension. Educators should thus aim to discriminate between the ability of the student to read fluently (with appropriate decoding skills), while also understanding what is being read. For instance, students learning English as an additional language in the UK were found to score lower on comprehension tasks despite equivalent decoding skills due to a weaker knowledge of vocabulary (Burgoyne et al., 2009).

Conclusion and Suggestions for Further Research

This study has shown the importance of caution in the interpretation of performance by Maltese bilingual students on English literacy tests developed and standardised for monolingual populations. This is particularly the case for English reading comprehension tests which entail a complexity of competencies beyond merely word decoding skills. The significant discrepancies between the scores obtained by Maltese students and the expected scores according to the NARA II British norms, as well as the discrepancies between their Reading Accuracy and Reading Rate and Reading Comprehension scores suggest that there is a need for more investigation of the confounding of word decoding skills with linguistic and cultural aspects of texts (Keenan, 2016; Peer and Reid, 2016; Oakhill et al., 2014). Moreover, in assessment of children’s reading comprehension development and difficulties, there is a need for assessors to consider the influence of home language background.

This study had its own limitations. It is noted that the sampling was limited not only in number but also in its attempt to represent average students rather than the whole range of student ability in reading comprehension skills. Thus, for instance, it did not represent the usual difference in literacy skills between males and females. It also reflected the field practice of including among the participants some students who were older (14 years) than the ceiling of the NARA II norms (13 years). Despite these limitations, however, the findings generally concurred with the norm discrepancies found in attempts at standardising UK English cloze reading tests (Liberato Camilleri, personal communication) as well as in Agius’s (2012) study of both single word reading and reading comprehension development in Maltese students. The present study also had the added value of relating the findings to the participants’ self-reported home language background.

Given the findings of this study and the performance of Maltese students on the PISA (2013; 2016) international assessment of literacy, the Maltese educational system needs to investigate more thoroughly the developmental profile of literacy skills of Maltese students with a goal of understanding clearly the challenges they face in the local bilingual context for the development of appropriate reading comprehension competencies. For instance, one area that requires investigation is the relation between both the methods of teaching as well as of assessment of reading comprehension at different levels of the education system. No less important is the need to

clarify, through the development of locally contextualised and standardised assessment instruments, the profile of skills and needs of students experiencing difficulties in any of the skills that relate to reading comprehension in both Maltese and English.

Acknowledgements

The research work disclosed in the publication is partially funded by the REACH HIGH Scholars Programme – post doctoral grant and by the European Union Operational Programme II – Cohesion Policy 2014-2020 ‘Investing in human capital to create more opportunities and promote the wellbeing of society’ European Social Fund.

References

Abedi, J., Huie Hofstetter, C., & Lord, C. (2004). Assessment accommodations for English language learners: Implications for Policy-Based Empirical Research. Review of Educational Research 74, 1-28. Agius, R. (2012). The development of a literacy assessment battery for Maltese children. PhD dissertation. University of Malta. Bartolo, P.A. (1988). Maltese Word Reading Test. Test Construction Unit: Education Department, Malta. Basterra, M., Rosario, Trumbull, E., & Solano, G. (Eds) (2011). Cultural Validity in Assessment: Addressing Linguistic & Cultural Diversity. New York: Routledge. Burgoyne, K., Kelly, J.M., Whiteley, H.E., & Spooner, A. (2009). The comprehension skills of children learning English as an additional language. British Journal of Educational Psychology, 79, 735-747. Burgoyne, K., Whiteley, H.E., & Hutchinson, J.M. (2013). The role of background knowledge in text comprehension for children learning English as an additional language. Journal of Research in Reading, 36(2), 132 – 148. Cain, K., & Oakhill, J. (2006). Assessment Matters: Issues in the measurement of reading comprehension. British Journal of Educational Psychology, 76 (4): 697 – 708. Chall, J. S. (1996). Learning to Read: The Great Debate. 3rd ed. New York: McGraw-Hill. Cline, T., & Shamsi, T. (2000). Learning Needs or Special Needs? Luton, UK: University of Luton. Accessed from https://www.naldic.org.uk/Resources /NALDIC/Research%20and%20Information/Documents/RR184.pdf Colenbrander, D., Nickels, L., & Kohnen, S. (2017). Similar but different: differences in comprehension diagnosis on the Neale Analysis of Reading Ability and the York Assessment of Reading for Comprehension. Journal of Research in Reading, 40(4 ), 403–419.

Cutting, L. E., & Scarborough, H.S. (2006). Prediction of reading comprehension: Relative contributions of word recognition, language proficiency, and other cognitive skills can depend on how comprehension is measured. Scientific Studies of Reading 10 (3): 277–299. Ehri, L. C. (2005). Learning to read words: Theory, findings and issues. Scientific Studies of Reading 9 (2): 167–188. Elliot, C. D. (1983). The British Ability Scales, Manual 1: Introductory Handbook. Windsor, UK: NFER-Nelson. Elbeheri G., & Everatt JM. (2016). Principles and guidelines in test construction for multilingual children. In L. Peer & G. Reid (Ed.), Multilingualism, Literacy and Dyslexia (2nd edition ed., pp. 49-60). Abingdon: Routledge. European Agency for Special Needs and Inclusive Education (2014). Education for All. Special Needs and Inclusion in Malta. External Audit Report. Accessed April 2018: https://education.gov.mt/en/Documents/Special%20Needs%20 and%20Inclusive%20Education%20in%20Malta%20%C2%AD-%20External% 20Audit%20Report.pdf European Union High Level Group of Experts on Literacy (2012) Act now! Final report: EU High Level Group of Experts on Literacy. Luxembourg: Publications Office of the European Union. Everatt, J., & Reid, G. (2010). “Motivating children with dyslexia.” In J. Fletcher, F. Parkhill, and G. Gillon (Eds.), Motivating Literacy Learners in Today’s World (pp. 67–78). Wellington: NZCER Press. Everatt, J., Smythe, I., Adams, E., & Ocampo, D. (2000). Dyslexia screening measures and bilingualism. Dyslexia 6 (1): 42–56. Falzon, J. (1972). Multilevel Group Reading Test. Malta: Education Department. Figueroa, R. A. (1989). Psychological testing of linguistic-minority students: Knowledge gaps and regulations. Exceptional Children 56 (2): 145–152. Frith, U. (1985). “Beneath the surface of developmental dyslexia.” In K. Patterson, J. Marshall, and M. Coltheart (Eds.), Surface Dyslexia: Neurological and Cognitive Studies of Phonological Reading (pp. 301 – 330). Hillsdale, NJ: Lawrence Erlbaum. Goswami, U. (2000). Phonological representations, reading development and dyslexia: Towards a cross-linguistic theoretical framework. Dyslexia 6 (2): 133 – 151. Gough, P. B., & Turner, W. (1986). Decoding, reading and reading disability. Remedial and Special Education 7(1), 6–10. Ghirxi, J. (2013). Exam Access Arrangements in Malta: Consultation Document Prepared for the Ministry of Education and Employment on Identified Current Challenges and Proposed Ways Forward. Accessed April 2018 from https://curriculum.gov.mt/en/resources/further-resources/documents/ eaa_consultation_document.pdf Grech, L., Everatt, J., Bartolo, P., & Camilleri, L. (2017). Developmental variation in predictors of reading comprehension amongst Maltese students. Malta Review of Educational Research 11 (1): 3–27.

Hagley, F. (2002). Suffolk Reading Scale 2. London: GL Assessment. Hutchinson, J. M., Whiteley, H. E., Smith, C.D., & Connors, L. (2003). The developmental progression of comprehension-related skills in children learning EAL. Journal of Research in Reading 26 (1): 19–32. Joshi, R. M., & Aaron, P. G. (2000). The Component Model of Reading: Simple View of Reading made a little more complex. Reading Psychology, 21 , 85-97. Keenan, J. M. (2016). Assessing the assessment. Perspectives on Language and Literacy, 42(2), 17–20. MATSEC Support Unit. (2016). SEC Examinations 2015 Statistical Report. http://www.um.edu.mt/__data/assets/pdf_file/0004/271273/SECReport2 015NewFormat.pdf McBride-Chang, C. (2014). Children’s literacy development. London: Routledge. Ministry for Education and Employment. (2015). Language Education Policy Profile – Malta. Strasbourg: Language Policy Unit, Council of Europe. https://www.coe.int/t/dg4/linguistic/Source/ProfileMalta_final2015_EN.pdf Ministry of Education and Employment. (1999). Creating the Future Together: National Minimum Curriculum. Malta: Ministry of Education. Malta http://www.ibe.unesco.org/curricula/malta/mm_alfw_1999_eng.pdf Mortimore, T. (2012). “The Big Lottery: Dyslexia and Multilingualism Project.” In J. Stein, and K. Saunders (Eds.), The Dyslexia Handbook 2012: A Compendium of Articles and Resources for Dyslexic People, their Families and the Professionals Who Deal with Them. Bracknell: British Dyslexia Association. Neale, M. D. (1997). Neale Analysis of Reading Ability Revised. Windsor, UK: NFER Nelson. Oakhill, J., & Cain, K. (2012). The precursors of reading ability in young readers: Evidence from a four-year longitudinal study. Scientific Studies of Reading , 16 (2): 91–121. Oakhill, J., Cain, K., & Elbro, C. (2014). Understanding and teaching reading comprehension: A Handbook. London: Routledge. O’Brien, B. A., & Wallot, S. (2016). Silent Reading fluency and comprehension in bilingual children. Frontiers of Psychology, 7: 1265. Peer, L., & Reid, G. (2016). Multilingualism, literacy and dyslexia: Breaking down barriers for educators (2nd^ ed.). Milton New York: Routledge. Programme for International Student Assessment: PISA 2009+ Malta Report. (2013). Ministry for Education and Employment, Malta. Programme for International Student Assessment: PISA 2015 Malta Report. (2016). Ministry for Education and Employment, Malta. Rasinski, T.V. (2006). A brief history of reading fluency. In S. Samuels & A. Farstrup (Eds.) What Research Has To Say About Fluency Instruction (p. 70-93). Newark, DE: International Reading Association. Sattler, J. M. (2008). Assessment of Children: Cognitive Foundations. 5th^ ed. La Mesa, CA: Jerome M. Sattler.

Shanahan, T, & Shanahan, C. (2012). What is disciplinary literacy and why does it matter? Topics in Language Disorders , 32(1), 7 – 18. Sireci, S. G. (1997). Problems and Issues in Linking Tests Across Languages. Educational Measurement: Issues and Practice , 16 (1), 12 – 19. Spooner, A. L. R., Gathercole, S.E., & Baddeley, A.D., (2006). Does weak reading comprehension reflect an integration deficit? Journal of Research in Reading, 29(2), 173 – 193. Strauss, C., & Quinn, N. (1998). A Cognitive Theory of Cultural Meaning. Cambridge: Cambridge University Press. University of Malta/Access Disability Support Committee & Ministry of Education/Directorate for Quality and Standards in Education. (2010). Reading Comprehension and Spelling Tests in Maltese and English. Malta: UM. University of Malta. (2015). Guidelines to MATSEC Examinations Access Arrangements. Accessed April 2017 https://www.um.edu.mt/__data/assets/pdf_file/ 0003/247224/Guidelines_2015.pdf Valdés, G., & Figueroá, R.A. (2004). Bilingualism and testing: A special case of bias Norwood, NJ: Ablex. Wainer, H., & Braun, H.I. (2013). Test Validity. New York: Routledge. Wheldall, K., & Arakelian, S. (2016). Comparing the performance of older lowprogress readers on the York Assessment of Reading for comprehension with performance on the Neale Analysis of Reading Ability and other measures of reading and related skills. Australian Journal of Learning Difficulties, 21(2) , 157166. Zheng, H. D. (2014). Evidence-based Practices: Reading comprehension instruction and teacher self-Efficacy. DEdPsy Thesis, UCL Institute of Education. http://discovery.ucl.ac.uk/10021726/1/Thesis_Library%20copies.pdf Ziegler, J., Bertrand, D., Toth, D., Csepe, V., Reis, A., Faısca, L., & Blomert, L. (2010). Orthographic depth and its impact on universal predictors of reading: A cross-language investigation. Psychological Science, 21(4), 551 – 559.