*Keywords:* Readability, LIX formula, PIRLS, Maltese

‘gabriella-mifsud’

---

Volume 13, No.1., 65 78 Faculty of Education©, UM, 201 9

The Readability of Maltese Examination Texts

Gabriella Mifsud Minsitry of Education, Malta Gabriella_Mifsud21@hotmail.com

Abstract: This paper discusses the notion of readability, and it introduces a readability formula that has been tested for its applicability on texts in Maltese. The LIX formula can be applied to both Maltese and English texts and it can therefore be used to compare the reading scores of the same group of learners sitting for the same assessment in the two languages. In this paper, the LIX formula is used to examine possible reasons for the contrasting results in reading comprehension in Maltese and English in the PIRLS international assessments carried out in Malta. The levels of readability resulting from the application of the LIX formula to the texts in the two languages clearly show that the levels of difficulty are not equivalent, and therefore no conclusions can be reached about the reading abilities of these cohorts tested in Maltese, when compared to those tested in English. Keywords: Readability, LIX formula, PIRLS, Maltese

Introduction

‘The more you read, the more things you will know. The more you learn, the more places you will go’ (Dr.Seuss). It is well known that reading is essential in every person’s life, especially children who need to read throughout their educational experience. Reading is not only important because it helps learners academically, but it also helps them develop creatively. In fact, reading ability is considered so important in education that it is commonly tested in language examinations. In this article, I discuss the issues that are relevant when choosing appropriate reading texts in Maltese for use with children, and in particular, in examinations. Educators need to be sensitised about the importance of choosing the right texts, the right material and the right books for children to read. If children do not read the right books that are suited for their age, they will stop reading (DuBay, 2004). Furthermore, if the text is too easy, the students will lose interest in the text itself (Carrell, 1987). It is important for educators, parents, publishers and so on to be able to establish the right reading level of a text for a given age. For this reason readability formulas were invented.

The concept of readability could be mistaken for literacy, however, when a text is readable, it means that a text has the “quality of being easy or enjoyable to read” (Thompson, 1998, p. 1142). On the other hand, literacy is related to the ability to read, and to the quality of a text as “being clear enough to read” (Thompson, 1998, p. 777). For a text to be good and clear for the reader, it needs to meet several criteria, such as, appropriate vocabulary for the age group, correct grammar, and the font of the text must be clear, legible and comfortable to read. The text also needs to be readable, which means that it needs to be suitable for the reader.

Over the years, many readability formulas were created. Readability formulas are necessary since the texts, and their authors, do not always indicate the age for whom such texts are intended (Carrell, 1987). In 1915, Thorndike published a book called ‘The Teacher’s Word book’ , in which he provided a graded guide on how to teach certain texts and how to recognize a good text for a particular age (Carrell, 1987). At this point, Lively and Pressey (1923) came up with the first readability formula (Lively & Pressey, 1923). Since then, over 200 readability formulas were created for different educational sectors (DuBay, 2004). Although there is a considerable choice of formulas, the choice of a readability formula depends on the context in which it is to be used because some formulas are meant for particular sectors or professions rather than academic domains. One of the advantages of using a readability formula is that there is no need for the reader’s participation to determine whether a text is difficult or not (Kasule, 2011). According to various studies, the difficulty of the text is determined by the frequency of the words, the median of long words, and how long the sentences are (Anderson & Davison, 1986). A number of readability formulas are introduced below.

Readability Formulas The Army’s Automated Readability Index (ARI) is a well-known readability formula. However, as the title itself indicates, this formula is used for texts which are related to the army. The formula is as follows:

– 21.43

(Media, 2018)

This formula differs from others because it is based on the number of letters in a word instead of the number of syllables. Another formula which has a specific use is the Power-Sumnerl Kearl. This formula is mostly used for primary school texts and therefore it cannot be used to examine texts for older readers. This formula is worked out by choosing a 100-word long text. The exact number of words is to be counted and then divided by the number of sentences in order to obtain the Average Sentence Length (ASL). Afterwards, the number of syllables has to be divided by the number of words to obtain

the Number of Syllables (NS). Once the user works these out, the information has to be inputted in the following formula:

RA = 0.0778(ASL) + 0.0455(NS) + 2.7971

The result, i.e. Reading Age (RA), is the exact age of the reader for which the text is intended. This is a very useful formula, however, it can only be used for the English language and for primary school level texts (Media, 2018). This is due to the fact that it relies on the number of syllables in a word, which differs greatly from one language to another. One of the most commonly used formulas for the English language is the Flesch formula. According to Flesch the two things that make a text difficult are the number of syllables per word, and the sentence length. In fact, these two variables are included in the Flesch formula:

RE = 206.835 – (1.015 x ASL) – (84.6 x ASW) RE – The difficulty of the text ASL – The average length of sentences (This is obtained by dividing the number of words by the number of sentences). ASW – The average length of syllables (This is obtained by dividing the number of syllables by the number of words).

The lower the result, the more difficult the text is. Although this formula is very popular, it leaves out various factors which are also considered to make a text difficult. Some words might seem long, but not all the long words are difficult for the students. For example, the word elephant is considered to be a long word because it is made up of 8 letters and 3 syllables. However, students are most likely to learn this word at a very young age and therefore it cannot be deemed a difficult word (Harrison, 1980).

The Dale-Chall formula avoids the above-mentioned problem. This formula provides the user with 3000 English words which are considered to be common, or used very frequently. Those words which are not part of the list are considered to be difficult words (DuBay, 2004). The more words in a text that are not present in the list, the more difficult the text is considered to be for children. This means that this formula also caters for those monosyllabic words which are deemed difficult, and it eliminates long words which are considered to be easy. After the user chooses the words which are not part of the list, the rest have to be worked out in the following formula:

Raw Score = 0.1579 * (PDW) + 0.0496 * ASL Raw Score – The difficulty of the text PDW – The percentage of difficult words ASL – The sentence average (This is obtained by dividing the number of words by the number of sentences). (Media, 2018)

Although it seems that this formula has a number of advantages due to the fact that it also considers the monosyllabic words which are difficult for students, it is not very practical because it requires a lot of time to check words which are not part of the extensive list that Dale-Chall presents.

Choosing a readability formula is important because teachers’ assumptions about the right readability level of a text for learners are not always correct. In fact, according to Harrison (1980), the teachers’ calculations might be mistaken by six or seven years. Therefore, the use of readability formulas is essential to check whether a text is appropriate or not for a particular age group. When it comes to the English language, it is quite easy to choose a readability formula because many formulas have been created for this international language. However, when it comes to a language like Maltese, the story is a little different because until recently a readability formula had never been created for this language. This is why I pioneered a study (Mifsud, 2018) in order to try and find a tool that would be suitable for adjudicating the readability of texts in Maltese. After some trials with a number of formulas, I chose the LIX formula. The LIX formula leaves space and opportunity to be used with various languages because it does not rely on the number of syllables. In fact, the LIX formula can be used both for Maltese and English and other Eastern European languages (Tillman & Hagberg, 2014). From the testing of texts I have carried out so far, I feel confident that the LIX formula can be applied to Maltese. In fact, I have also found it useful to compare the readability of examination texts in Maltese and English given to the same age group, to find out whether the texts in each of the languages were of equivalent difficulty. In what follows I will explain the implementation and results obtained from the PIRLS reading tests in Maltese and English using the LIX formula.

Tests of reading ability in Malta

Among the international measurements carried out to compare the reading achievement of students in various countries are the Trends in International Mathematics and Science Study (TIMSS) and the Progress in International Reading Literacy Study (PIRLS). These assessments test fourth grade (Yr 5 primary) students by using a number of reading comprehension texts in order to determine the children’s reading competence. The first test was carried out in Malta in 2011, and subsequently another test was carried out in 2016 in which fifty countries, including Malta, participated. Reading comprehension tests for these international assessments are written in English, and they are then translated into different languages so that all students can be assessed in their school language. However, as I argue below, the reading difficulty of texts varies from one language to another, depending on, for example, word length, morphological structure and vocabulary. In fact, as I will explain, the

translation of a text into Maltese rendered the same text more difficult. In order to compare readability between texts in different languages a simple translation does not do justice to the students sitting for the assessment. To be fair, an international reading test in various languages needs to establish readability criteria first, rather than simply provide a translation from English.

The Lasbarhetsindex (LIX) formula

As previously stated, the LIX formula can be applied to texts in different languages because it relies on the number of words, the number of sentences and the number of long words in the texts. Words are considered long if they contain more than 6 letters. Once all the information about a text is gathered, the user has to work out the following formula:

LIX = W/S + (L*100)/W W = The number or words in the text. S = The number of sentences in the text. L = The number of long words in the text.

Once the LIX formula gives the result, it should be interpreted as follows (Tillman & Hagberg, 2014, p. 5):

Result Text difficulty

< 25 Texts for children 25-30 Simple texts 30-40 Normal/fiction texts 40-50 Factual texts 50-60 Technical texts

60 > Difficult^ texts^ /^ research^ texts^ / dissertations Table 1: Readability levels based on the LIX formula

The formula is able to give a scientific number of readability for a text that can be in either Maltese or in English. Thus, I will compare the readability of the English texts with that of the Maltese texts given in the PIRLS 2016 assessments. The results will not only provide the readability, or level of difficulty for the texts in each language, but it also allows for a comparison between the texts in Maltese and English language.

However, before I move to the application of the formula a couple of caveats are in order. Table 1 gives a general idea of the category which the text falls under, but the formula leaves out other fundamental details. For example, although a text might give a readability result of 25 and under, which means that it is considered as a children’s text, it still does not specify the exact age

of the children with whom the text can be used. For instance, 5 year olds are considered as children but so are 9 year olds. This means that the formula only gives a general idea of age appropriateness. Another disadvantage of the LIX formula is that the result it gives is nothing more than ordinal scales (Musvoto & Gouws, 2009). For example, if a text gives a result of 15 readbility difficulty, and another one which gives an 18 readability difficulty, it is clear that the latter text is more difficult than the first and the difference between both is a readability difficulty of 3. However, if you have a text with a readability difficulty of 35 and another one of 38, undoubtedly the 38 readability text is much more difficult than the first one. However the difference between both comparisons is a readability difficulty of 3. This means that the values obtained by the readability formula are nothing more than ordinal scales which means a “relationship always look the same irrespective of the angle from which it is viewed” (Musvoto & Gouws, 2009, p. 430).

As mentoined earlier, the LIX formula is based on the visual elements in a text rather than on the technical syllabic structure and morphology of the text. This shows that this formula is better suited to Maltese than other formulas previously discussed. Since it is based more on the visuals, the għ and the ie , which in Maltese are considered as one letter, in the application of this formula they are considered as two because the reader has to process two alphabetical signs. Another factor which is taken into consideration when working with this formula is the definite article. The definite article in English is considered as a separate word. Although technically the definite article in Maltese is not considered as a stand-alone word (Borg & AzzopardiAlexander, 1997), it must be considered as such when appliying the LIX formula because the reader needs to process it, first visually and then semantically. For example, “The car is green”, is translated as “Il-karozza hija ħadra.” In both sentences the article defines the reference to a particular car, hence the reader has to understand that the writer is not referring to just any car.

The LIX formula considers a word to be long if it has 7 or more letters, but then it does not distinguish further between a word with 7 letters and one with 15 letters. The problem is that the 15-letter word definitely poses a bigger challenge for the reader. While in English 15 letter words are rare, in Maltese, due to its morphological structure, 15 letter words are common. The formula also fails to take into consideration those words which in spite of their length might be easy for the children because they would be familiar with them. While the formula gives a scientific idea of level of readability, it does not take into consideration the social context of the reader, and therefore which vocabulary might be easier for them (Biesta, 2007). Furthermore, the frequency of the same words in the text also plays a fundamental role when it

comes to judging the difficulty of a text (Carrell, 1987). Hence, when applying the LIX formula it is important to factor in these textual features.

An analysis of the PIRLS texts used in Malta

The PIRLS test is carried out every five years. It was conducted in Malta for the first time in 2011 and the Maltese children sat for both the English and the Maltese reading tests. In 2016 the Maltese children sat only for the Maltese test. A total of 3647 Maltese students participated in the PIRLS study in 2016, where 1754 were females while 1893 were males. The age of the participants was around 9.7 years and they were selected randomly from 95 primary schools. Two thousand and thirty-three (2033) students were selected from 62 State schools, 1245 students were selected from 25 Church schools and 369 students were selected from eight Independent schools (Ministry of Education). Malta’s mean reading score (452) was significantly lower than the international average (500) and was ranked 40th out of 50 participating countries in 2016. The mean reading score of Maltese students in the PIRLS 2016 Maltese main test (452) was significantly lower than the mean reading score in the PIRLS 2011 Maltese benchmark test (457) and the PIRLS 2016 English main test (477) (Ministry of Education). Post-test publications blame this lack of achievement on the “socio-economic and social background” of the test-takers (Ministry of Education, p. vii). Although this might be one of the reasons for low scores, in this article I would like to explain that there are also important issues related to the readability of the texts themselves.

In the next section I will compare the level of readability of each of the texts in Maltese and English in order to examine the issues that seem to be impacting on such an unhappy result for the Maltese participants in the PIRLS 2016 assessment. This will include the LIX result, together with other factors, which, as discussed before, are not taken into consideration by the LIX formula.

PIRLS comprehension paper 1: Macy and the red hen / Mary u t-tiġieġa lħamra

This reading comprehension text was 6 pages long, which is quite long for a 9 year old child. Students get frustrated and panicked as soon as they see a 6page long comprehension. The length of a text should be taken into consideration when assessing students, not just Maltese students but also those participating in other languages. Furthermore, the Maltese version had several long words. This required in a high level of reading ability as also shown in the result when the LIX formula is applied. In fact, the Maltese version obtained a result of 34.59 which is alarmingly high when compared to the 23.29 readability level of the English version, according to the LIX formula. This major difference comes from the fact that the Maltese version

has many longer words than the English version: the English version has 85 long words while the Maltese version has 190, almost double the number. Furthermore, when considering that the formula does not distinguish the difficulty between words that are longer than 7 letters, an additional difficulty for the Maltese text has to be included. In fact, the longest word in the Maltese text is made up of 12 letters, tibbossjahom (Anderson P. , 2016, p. 3), while the longest words in the English version never exceed the 9-letter count. This already shows that the students sitting for the English version are in a favourable position when compared to those sitting for the Maltese version.

Another factor which the Maltese students had to deal with, unlike their English counterparts, were problems emanating from the translation itself. The Maltese version included a number of newly transliterated words. For example, instead of tibbossjahom , the translator could have easily written tikkmandahom. One must keep in mind that the test-takers are only 9 years old, so that even if in an adult environment employees are likelty to use words like ‘boss’, ‘bossy’ and ‘jibbossja’, these are probably not part of the 9 year olds’ sociolinguistic context.

Throughout the text, a number of words were translated individually without attention to the linguistic context. In the first paragraph of the English text, one finds the following phrase, “…hens exploded into the yard,” (Anderson P. , Macy and the Red Hen, 2016, p. 3) which is translated as, “…tiġieġ splodew għal ġol-bitħa” (Anderson P. , Mary u t-Tiġieġa l-Ħamra, 2016, p. 3). The literal translation of the word exploded is splodew , however in Maltese, this word is not normally used in this context. Instead of splodew the translator should have used phrases like ħarġu bis-saħħa or ħarġu f’salt and these would have been more appropriate in the context, and more easily understood. A similar example is found in the last paragraph. In English, “terrible wings” (Anderson P. , Macy and the Red Hen, 2016, p. 8) is translated as “ġwienaħ terribbli” (Anderson P. , Mary u t-Tiġieġa l-Ħamra, 2016, p. 8). Once again, this literal translation does not fit the context. Instead of ġwienaħ terribbli, ġwienaħ tal-biża’ would have made more sense in Maltese. These are a few examples of how a translated text poses greater challenges to the test-takers sitting for the test in other languages than the original version.

One of the limitations of the LIX formula is that it does not calculate the number of repeated words in the text. As Carrell stated, whenever the same words are repeated in the same text, the reader becomes more familiar with them, making it easier for the reader to pronounce them, read them and understand them (Carrell, 1987). This limitation should not be ignored, especially when in a single text a word is repeated 24 times. In the Maltese text the word tiġieġa is written 24 times. Moreover, even though the word falls under the criterion of long words according to the LIX formula, the word

tiġieġa is quite a common word in the Maltese language and students normally learn this word in their early years. However, apart from the word tiġieġa no other word is repeated more than 4 times and therefore the text still remains relatively difficult when compared to the English one.

PIRLS comprehension paper 2: Leonardo Da Vinci: a man ahead of his time / Leonardo Da Vinci: bniedem għaref ħafna għal żmienu

The second text that the students had to sit for was about Leonardo Da Vinci. The text was very informative and quite interesting to read, but one should always keep in mind that this topic does not necessary appeal to 9-year-old students sitting for an international assessment on reading ability. According to Anderson and Davison (1986), when students are familiar with the subject, they have a tendency to perform better, and obviously the opposite holds true.

The readability level of this English text is 32.55 while the Maltese paper obtains a much higher score of 38.96 readability level. What is more worrying is that, according the LIX readability formula, texts between 30-40 readability level, fall under the category ‘normal texts’. i.e. a level higher than simple texts. Should students, at this age, be sitting for normal texts or should they be sitting for something easier, more appropriate, for their age? Maybe the texts can be considered as slightly easier because, in both texts, the word Leonardo is repeated 13 times. Moreover, the word Leonardo is a name and hence there is no complexity in understanding the word. However, the name does not alter the LIX result in any significant way. This can be confirmed because not many long words are repeated in either text. Once again, the Maltese PIRLS paper has more long words (129), than the English paper ( 95 ), and this makes the Maltese version more difficult. Moreover, the longest word in the English PIRLS paper consists of 11 letters, “constructed” (Leonardo da Vinci a man ahead of his time, 2016, p. 6) , while the Maltese paper has a 14-letter long letter, “invenzjonijiet” (Leonardo da Vinci bniedem għaref ħafna għal żmienu, 2016, p. 6). It is clear that a question arises about the criteria used by the exam setters to decide on the choice of texts, and whether they had established any readability criteria at all.

In this text, the Maltese translation is more flowing than the previous one, and the sentences are translated according to the context rather than word for word. However, there is an error in the first page of the Maltese text. The English sentence:

“He also learned how to mix different types of colours to make paints, and how to use metal to make sculptures.” (Leonardo da Vinci a man ahead of his time, 2016, p. 5)

is translated into Maltese as:

Tgħallem ukoll kif iħallat tipi ta’ kuluri differenti biex joħloq żebgħat differenti, u kif juża l-metall biex joħloq l-iskulturi.” (Leonardo da Vinci bniedem għaref ħafna għal żmienu, 2016, p. 5)

The word żebgħat in Maltese does not exist because żebgħa is a collective noun, and it takes an adjective in the singular. Hence the correct translation should have been:

Tgħallem ukoll iħallat tipi ta’ kuluri biex joħloq żebgħa differenti, u kif juża lmetall biex joħloq l-iskulturi.

On the whole, in spite of the fact that this text was shorter with few long words, judging by the result of the LIX formula, both the Maltese and English versions were beyond the capability of the students. Furthermore, the Maltese text clearly required a higher reading ability than the English version.

PIRLS comprehension paper 3: Flowers on the Roof / Fjuri fuq il-Bejt

The level of difficulty of the English version of this text was the easiest of the four given in 2016, but the same cannot be said for the readability level of the Maltese translation. The story is not that difficult and it is enjoyable to read because it is a narrative. Translation of names, can, however, pose problems. In the English comprehension the grandmother is called Gunnjona while in the Maltese paper she is called Ġorġina. One also wonders on the choice of Mary (why not Maria?) as a translation for the name Macy in the text ‘Macy and the red hen / Mary u t-tiġieġa l-ħamra’ discussed above. Criteria for the translation of names should also be established in international assessments like the PIRLS.

This comprehension test is four pages long. One has to keep in mind that these students have to answer questions related to the text and the longer it is, the harder it will be for the students to look up the answer. Apart from these challenges, the Maltese students had to face a 27.47 readability level, while the English students sat for an 18.95 readability level text. This shows the huge disadvantage that Maltese students faced when compared to those sitting for the English test.

Let us look at readability factors. While the English text has only 70 long words, the Maltese version has almost double that number ( 133 long words). Moreover, the longest word in the English paper is made up of 11 letters, that being “countryside” (Sigurdardӧttir, Flowers on the Roof, 2016, p. 22). On the other hand, the Maltese paper has a 13 letter word, “fortunatament” (Sigurdardӧttir, Fjuri fuq il-Bejt, 2016, p. 20). It must also be noted that

although the English version has 70 long words, most of these words are made up of 7 or 8 letters. On the other hand, the majority of the long words in the Maltese text are 10-letter words.

In this particular text, the translation is quite clear but there are some syntactical errors which are evident throughout the whole paper. For example:

“Anke jien kont ħafna eċċitat!” “Huma wisq kbar !” (Sigurdardӧttir, Fjuri fuq il-Bejt, 2016, pp. 20, 21)

Although the sentence is understandable, the correct way of translating these sentences would be:

“Anke jien kont eċċitat ħafna!” “Huma kbar wisq!”

It is fundamental for a translation to be correct not only in terms of content, but also in terms of grammar. It is unacceptable for test-takers to have to deal with incorrect aspects in a text, especially when they are being tested and placed in an international contest.

PIRLS comprehension paper 4: The Green Sea Turtle’s Journey of a Lifetime / Ilvjaġġ Tul il-Ħajja tal-Fekruna l-Ħadra tal-Baħar

This is the second easiest text of the English texts, and the the easiest one in Maltese. Hence, one can already see that the results cannot be considered as reliable since the texts have a different level of readability.

Although the readability level of Maltese and English are not very high, that of 26.41 and 22.15 respectively, I felt that this was the hardest text of them all. These texts are written in the present tense, and the use of the imperfett rather than the past tense ( perfett ) in Maltese rendered the text somewhat unclear. The information in the text is scientific and it describes the life cycle of a turtle. The text is also extremely long. Both texts almost amount to a 1000word comprehension test each. Students will definitely get tired reading such a text especially because it contains a huge amount of information.

The reason for obtaining a low readability level in a text also depends on the number of long words in comparison with the length of the whole text. For the first time, the longest word is found in the English text because the word “unfortunately” (Miller, The Green Sea Turtle’s Journey of a Lifetime, 2016, p. 12) is made up of 13 letters. This was translated into Maltese as b’“xorti ħażina” (Miller, Il-Vjaġġ Tul il-Ħajja tal-Fekruna l-Ħadra tal-Baħar, 2016, p.

12). In fact, in this case, although there are more long words in the Maltese text, there are also several long words in the English text that are made up of 9 letters or more, such as hatchling, continues and scientist. The longest Maltese word, jipproteġiha , is made up of 12 letters.

In this text there are also some poorly translated sections, and the title itself is an example of this. The Green Sea Turtle is the name of a species. In Maltese the literal translation for turtle is fekruna tal-baħar and so the name of the species was translated literally as though it was a common noun. The same applies to the word “flippers” (Miller, Il-Vjaġġ Tul il-Ħajja tal-Fekruna lĦadra tal-Baħar, 2016, p. 11). This is a well-known English word, and there is no equivalent for it in Maltese because Maltese people use “flippers” as well. When English words are borrowed in Maltese they are normally italicised. However, the word flippers in the Maltese text was not written in italics, and it is not clear to what extent this might have required longer processing time for the Maltese test-takers.

Throughout the text there are other unclear translations. For example, a particular sentence reads, “Their memory of chemicals or odors in the water also may help them find their way” (Miller, The Green Sea Turtle’s Journey of a Lifetime, 2016, p. 14), which is translated as, “Il-memorja tagħhom ta’ kimiċi jew irwejjaħ tal-baħar tista’ wkoll tkun t’għajnuna għalihom biex isibu triqithom” (Miller, Il-Vjaġġ Tul il-Ħajja tal-Fekruna l-Ħadra tal-Baħar, 2016, p. 14). The Maltese version means that the turtles were chemists, but what the writer meant was that the chemicals found in the sea might help the turtles. This translation should have been: il-kimiċi li jinsabu fil-baħar jistgħu jgħinu lil dawn il-fkieren. Although most of the English words are translatable into Maltese, the meaning and/or the connotations might not always be exactly the same. This is clearly seen in the sentence, “Using her front flippers, she digs a wide pit,” (Miller, The Green Sea Turtle’s Journey of a Lifetime, 2016, p. 15) which is translated as “Permezz tal-flippers ta’ quddiem tħaffer fossa wiesgħa” (Miller, The Green Sea Turtle’s Journey of a Lifetime, 2016). In Maltese, the meaning of fossa directly translates to cesspit and hence the context is highly different. Therefore, the right translation should have been tħaffer ħofra wiesgħa which directly translates into digging a whole. In this text there are other mistranslations. For example, an English phrase reads, “…which could last up to 80 years” (Miller, The Green Sea Turtle’s Journey of a Lifetime, 2016, p. 15). This is translated as, “…tista’ ttul sa 80 sena” (Miller, Il-Vjaġġ Tul il-Ħajja tal-Fekruna l-Ħadra tal-Baħar, 2016, p. 15). The word ttul in Maltese refers to length and not to longevity, and could have very easily confused the students.

Conclusion

In this article I have highlited a number of deficiencies in the way the PIRLS 2016 reading tests were translated into Maltese. Although there are certain aspects of reading difficulty which the LIX formula does not cater for, it is still a useful tool giving a good indication about the appropriateness of a text for a certain age group. In order to obtain a fair and comparable result of readability, a fair and comparable text must be used. Unfortunately, as I have shown, this was not the case in the PIRLS 2016 texts. It is of fundamental importance that criteria of readability are established, including reference to vocabulary, context and the morphological and syntactic structure of a language, especially if a language test is being used for international comparison. In the absence of such criteria, and very poor translations, all the test-takers in other languages than English (because English was the original version and all other versions are translations) are being very unfairly and unreliably assessed.

References

Anderson, P. (2016). Macy and the Red Hen. PIRLS. Anderson, P. (2016). Mary u t-Tiġieġa l-Ħamra. PIRLS. Anderson, R., & Davison, A. (1986). Conceptual and Empirical Bases of Readability Formulas. Chicago: University of Illinois. Baker, E. (1993). The Impact of Readability Level on Questionnaire Measure Reliability. LIngilterra: Emerald Backfiles. Biesta, G. (2007). Why “What works” won’t work: Evidence-Based practice and the Democratic deficit in educational research. Educational Theory , 1-22. Borg, A., & Azzopardi-Alexander, M. (1997). Maltese. New York: Routledge. Carrell, P. (1987). Readability in ESL. In A. H. Urquhart, Readability in a Foreign Language (pp. 21-40). Stati Uniti: Southern Illinois University. DuBay, W. (2004). The Principles of Readability. California: Impact Information. Harrison, C. (1980). Readability in the classroom. Great Britain: Cambridge University Press. Kasule, D. (2011). Textbook readability and ESL learners. Francistown: University of Botswana. Leonardo da Vinci a man ahead of his time. (2016). PIRLS. Leonardo da Vinci bniedem għaref ħafna għal żmienu. (2016). PIRLS. Lively, B., & Pressey, S. (1923). “A Method for Measuring the ‘Vocabulary Burden’ of Textbooks. Educational Administration and Supervision Vol. 9 , pp. 389-398. Media, M. B. (2018, 11 28). Check your Readability. Retrieved from Readability Formulas: http://www.readabilityformulas.com/new-dale-chall-readability- formula.php Mifsud, G. (2018). Analiżi xjentifika tat-testi użati fil-Fehim il-Qari fil-livell tal-eżami talBenchmark. Malta: University of Malta. Miller, G. (2016). Il-Vjaġġ Tul il-Ħajja tal-Fekruna l-Ħadra tal-Baħar. In G. Miller, Turtle Travels. PIRLS. Miller, G. (2016). The Green Sea Turtle’s Journey of a Lifetime. In G. Miller, Turtle Travels. PIRLS.

Ministry of Education, a. E. (n.d.). Progress in International reading literacy study (PIRLS) 2016. Malta. Musvoto, S., & Gouws, D. (2009). The Concept of a scale in accounting measurement. South Africa: University of Pretoria. Sigurdardӧttir, I. (2016). Fjuri fuq il-Bejt. PIRLS. Sigurdardӧttir, I. (2016). Flowers on the Roof. PIRLS. Thompson, D. (1998). The Oxford English Dictionary. Stati Uniti: Oxford University Press. Tillman, R., & Hagberg, L. (2014). Readability algorithms compatibility on multiple languages. Stockholm: School of Computer Science Kungliga Tekniska Högskolan.