This study aimed to evaluate “High–Intermediate 3” (Iran Language Institute, 2008) and “New Headway Upper–Intermediate” Soars¹ in terms of their readability levels. To this end, the readability of ten and 12 reading passages of the two textbooks taught to the students of English as a foreign language (EFL) and English as a second language (ESL) at the same level of language proficiency, respectively, was analyzed via the Flesch Reading Ease Score (FRES) and Coh–Metrix Easability Score (CMES) consisting of narrativity (NAR), syntactic simplicity, word concreteness, referential cohesion, deep cohesion, verb cohesion, connectivity, and temporality components. The results showed although the FRESs are developed on reading passages written for native speakers of English, they correlate very highly not only with the CMES but also with its NAR component obtained on the passages written for ESL students, indicating that the FRESs are valid readability measures of ESL texts as well. The FRES and CMES did not, however, correlate significantly with each other on EFL texts, indicating that the EFL and ESL texts differ from each other in terms of readability. The results are discussed from the perspective of micro structural approach to schema theory and suggestions are made for future research.

Keywords: readability, texts, co–metrix, english as a first, second and foreign language

EFL, english as a foreign language; ESL, english as a second language; FRES, flesch reading ease score; CMES, coh–metrix easability score; NAR, consisting of narrativity

Readability is defined as the "comprehensibility of written text”.^2,3 It has been measured by various formulae developed for materials written for native speakers of a language (L1), or materials designed for second and/or foreign language students (L2). According to Brown et al.,⁴ being used with L1 texts, readability formulae provide a numerical index that estimates the reading difficulty of texts for native readers. Or, as Danielson⁵ put it “A readability formula is usually a mathematical equation that strives to relate the comprehension of the reader and the linguistic characteristics of the text.”

There have been a fairly large number of formulae developed to be used within L1 context. Examples of such formulae include the Flesch Reading Ease Index, the Fog Index, the Fry index, the Flesch-Kincaid Index, the Gunning Index, and the Gunning-Fog Index, to name some. All these indices are based on simple measures such as word length and sentence length.⁶ As a result, they have been widely criticized in the literature.⁷ Furthermore, few studies have attempted at developing L2 readability formulae Ozasa et al.⁸ & Gilliam et al.,⁹ or applying L1 formulae to L2 contexts.^9-11 Results reported about applying L1 formulae to L2 texts have not, however, been consistent.

A recently developed L2 index which takes into account cohesion, coherence and deeper levels of text processing is Coh-Metrix.¹² Graesser et al. ¹³ contended that Coh-Metrix provides analysis of text difficulty at multiple levels. Elfenbein¹⁴ enumerated some other merits for Coh-Metrix, chief among them is being user-friendly and free. For these reasons the present study attempts to apply it to the reading passages of two L2 textbooks used in English language institutes in Iran and explore its relationship with Flesch Reading Ease indices, i.e., The ILI English series: High-Intermediate 3 Iran Language Institute¹⁵ and “New Headway Upper-Intermediate”.¹

Literature review

The issue of readability of materials has been a topic of research for a long time. It has been defined from various perspectives in the literature. Dale et al.¹⁶ and DuBay,¹⁷ for example, defined it as the degree to which readers are successful in reading and understanding a printed material which they find interesting, at an optimal speed. As is evident in the definition, most researchers have associated readability with how much difficulty a text causes for its readers to understand it. Readability indices developed for L1, therefore, provide “a numerical scale that estimates the readability or degree of reading difficulty that native speakers are likely to have in reading a particular text”.⁴

Palmer¹⁸ and Ozasa et al.⁸ emphasized the need of an ideal course to have learning materials graded into appropriate stages each enabling the student to assimilate and use the language material. He also emphasized the important role of gradation of vocabulary in the students’ progress. Building on the argument, Ozasa et al.⁸ asserted that a pre-requisite for such gradation of materials is to be able to compare the readability of texts. As a result readability formulae provide a basis for the selection of texts. Using readability formulae, one can avoid the mismatch between the readers’ current level of proficiency and the one demanded by the text.¹⁹ Looking at the issue from another perspective, Zamanian et al.²⁰ added that these formulae also help authors write suitable texts for their intended audience.

Having established the need and reason for using readability formulae, the present researchers introduce the most widely used formulae developed to measure readability of texts by dividing them into two groups: the formulae developed for use in L1 context, and those developed for the L2 context. Most of them, however, deal with the former than the latter, particularly with texts in English. The methods discussed below all provide an estimate of how many years of education the reader needs to understand the text.^17,21 They are mostly based on simple measures such as word length and sentence length Crossley⁶ and DuBay,¹⁷ assuming that shorter words and sentences are easier to read and comprehend.⁸

L1 formulae

The flesch reading ease score: The Flesch²² Reading Ease Score (FRES) is one of the most commonly used readability indices. It can easily be computed in Microsoft Word, whenever a spelling and grammar check is performed by Microsoft Word, if the option is enabled in the software. The formula for the FRES is 206.835 – (1.015 x ASL) – (84.6 x ASW). The obtained score ranges between 0 (difficult) to 100 (easy), with 30 = very difficult and 70 = suitable for adult audiences. The ASL and ASW stand for “average sentence length (the number of words divided by the number of sentences)” and “average number of syllables per word (the number of syllables divided by the number of words)”, respectively.

The flesch-kincaid grade level: The Flesch-Kincaid Grade Level Kincaid et al.²³ [henceforth FKGL] rests on using ASL and AWL as in the FRES. It does, in fact, change the FRES into the U.S. grade school level ranging from 1 to 17. The formula to determine the FKGL is 0.39 (total words/total sentences) + 11.8 (total syllables/total words) – 15.59. Flesch [24] himself, however, equates his FRESs with grades as follows: 90 to 100 (5th grade), 80 to 90 (6th grade), 70 to 80 (7th grade), 60 to 70 (8th and 9th grade), 50 to 60 (10th to 12th grade (high school), 30 to 50 (college), 0 to 30 (college graduate).

The gunning-fog index: The Gunning-Fog index Gunning²⁵ has gained popularity “because of its ease of use. It uses two variables, average sentence length and the number of words with more than two syllables for each 100 words”.²⁰ It is based on the simple formula of determining Grade Level through multiplying 0.4 by (Average Sentence Length + Number of words more than two syllables long). It yields an index which ranges from 6 (sixth grade) to 17 (college graduates), each corresponding to the grade level needed for understanding the text.

The fry index: The Fry²⁶ readability index also relies on the average word length and the average sentence length. To find the grade level, one should check a graph consisting of a vertical axis representing the average length of sentences per 100 words, and a horizontal axis representing the average number of syllables per 100 words. The point that corresponds to the two values on the vertical and horizontal axes provides the suitable grade level of the text.

L2 formulae

Ozasa-fukui year level: Compared to researches dealing with L1 readability formulae, few studies have aimed at testing the efficiency of readability formulae developed for L1 to be used for assessing the readability of L2 texts.^4,8-11,27,28 Most studies have employed L1 formulae and have reported mixed results. Ozasa et al.,⁸ for example, applied the Flesch and Flesch-Kincaid formulae to estimate reading difficulty of English textbooks used in Japan. They found them as inaccurate representations of the features of the textbooks they analyzed. They reported that these indices could not differentiate difficulty level of the sentences of each textbook. Instead of using these indices, they proposed Ozasa-Fukui Year Level index to be used to assess readability of English textbooks used in Japan.

Mean scores on cloze tests: Brown¹⁰ also studied readability level of English texts within Japanese EFL context. As an EFL difficulty index, he used the mean scores of respondents to cloze tests and transformed them into percentile z scores. The findings of his study indicated that 6 first language readability indices investigated in that study, i.e. Flesch, Flesch-Kincaid, Fry, Gunning, Fog, and Gunning-Fog, all correlated highly with each other implying that all of them measured the same thing. On the contrary, those L1 indices had low correlations with the EFL difficulty index used in the study. This led Brown to conclude that L1 indices were not good predictors of EFL text difficulty. He suggested a combination of four features of text, all taken together, as accounting for a large percent of variance in EFL difficulty index. These features include a) number of syllables per sentence; b) average frequency of the deleted word in the rest of the passage; c) percent of words equal to or more than 7 letters in length; and d) percent of function words.

Brown et al.⁴ also used the cloze procedure with Russian students and used the mean of the students’ performance as indicator of the passage difficulty. Alongside the students’ mean performance, the same L1 readability indices as used in the 1998 study were used. The result was that L1 indices correlated moderately to highly with each other, but the correlation between Russian students’ mean scores and L1 indices were low, concluding that: “This lack of relationship could be due to any of the following: (a) that these L1 readability estimates are fine indicators of passage readability for native speakers but not for Russian EFL learners; (b) that the cloze passages are measuring something different from the simple readability measured by the L1 indexes; (c) that the Russian EFL learner’s scores on the cloze passages are measuring something much more complex than simple readability something like the students’ overall proficiency levels rather than the reading difficulty of the passages.⁴”

The same as Brown,¹⁰ Brown et al.,⁴ Greenfield²⁹ and Greenfield³⁰ used cloze procedure with Japanese university students. The reading passages were academic texts used in an earlier study by.^30,31 The readability formulae used in his study were Flesch and Flesch-Kincaid. The Pearson correlation between each of these L1 formulae and the mean cloze scores were found to be high. Thus, contrary to Brown, he concluded that traditional formulae could predict difficulty level of Japanese EFL texts well. The same results were reported in Greenfield³⁰ in favor of using L1 formulae in EFL contexts. The formula developed by Greenfield is called the Miyazaki EFL Readability Index and is scaled for EFL learners through the formula below:

Miyazaki EFL Readability Index = 164.935 - (18.792 × letters per word) - (1.916 × words per sentence).

Readability of spanish texts: In an attempt to develop a readability formula for elementary level materials in Spanish, Gilliam et al.⁹ adapted the Fry graph to measure the readability of Spanish texts at primary grades of 1, 2, and 3. In another study, Crawford¹¹ used the average sentence length and number of syllables per 100 words as two predictive variables of text difficulty level. Passages were selected from 10 elementary level reader series used in three Spanish speaking regions, i.e. the United States, Latin America, and Spain. A readability formula and graph was developed by him, the formula being presented below:

Grade level = [number of sentences per 100 words* (-.205)] + (number of syllables per 100 words* .049) -3.407

Readability of vietnamese text: Nguyen et al.²⁷ developed a readability formula for Vietnamese passages based on two factors of average word length, and average sentence length. The formula classified passages from 1 to 15, i.e. from texts suitable for elementary grades to college third year and above. These three variables predicted only %74 of the variance in the readability. Therefore, to compensate for the remaining %26, they revised the formula in 1985. They replaced average word length with the average of compound Sino-Vietnamese words in the passage. The second predictor, however, remained intact. The new formula was found to account for %90 of the variance and classified passages on a 12-level scale, based on the Vietnamese educational system, with passages rating 13 or above considered as college level. A and B represent the two formulae developed in 1982 and 1985, respectively.

A) Readability Level = 2 Word Length + .2 Sentence Length - 6
B) Readability Level = .27 Word Length + .13 Sentence Length + 1.74

Coh-metrix: Coh-Metrix was developed as a measure of readability at the University of Memphis. It measures cohesion and difficulty of texts at various levels of language, discourse, and conceptual analysis.⁶ The aim of developing this measure was to move beyond simple predictors of readability such as word and sentence length.¹⁹ It provides measures at five discourse levels of words, syntax, text base, situation model, and genre.^32,19 Such a deeper level analysis is made possible by synthesizing progress in a number of disciplines, such as psycholinguistics, computational linguistics, corpus linguistics, information extraction, information retrieval, and discourse processing.¹²

The analysis can be done online by simply entering the text on the website of Coh-Metrix available at www.cohmetrix.com. Coh-Metrix provides a measure of readability and Easability. It is measured on nine different areas each consisting of a number of indices, i.e., descriptive indices, referential cohesion, latent semantic analysis, lexical diversity, connectives, situation model, syntactic complexity, syntactic pattern density, and word information. A large number of indices are, however, produced in the output file under these nine categories. Measures of readability can be accessed through http://tool.cohmetrix.com/. The output generated by this tool provides 106 indices in the 3rd version of the software. The indices are categorized into nine measures. As it takes too much space to explain all these indices in detail here, the interested readers are referred to.³³

Crossley et al.³⁴ employed the data set used by Greenfield,²⁹ which were 31 academic texts, and investigated the efficiency of Coh-Metrix L2 Reading Index in measuring text difficulty. They incorporated three predicting variables of CELEX Word Frequency, Sentence Syntax Similarity, and Content Word Overlap. It was hypothesized that these variables more accurately reflected the cognitive processes involved in reading L2 passages. The results of the multiple regression analysis indicated that the combination of these three predictors accounted for 86% of the variance in the performance on cloze tests. Comparing the results with those produced by Flesch and Flesh-Kincaid proved Coh-Metrix to predict text difficulty level more accurately. The Coh-Metrix L2 Reading Index reported by Crossley et al.³⁴ and Crossley et al.³⁴ is as follows:

Predicted Cloze Score = - 45.032
+ (52.230 * Content Word Overlap Value)
+ (61.306 * Sentence Syntax Similarity Value)
+ (22.205 * CELEX Frequency Value)

Besides Coh-Metrix L2 Reading Index, the Coh-Metrix website provides the Coh-Metrix Easability Assessor, too. McNamara et al.⁹ conducted a Principal Components Analysis on 54 of the indices provided by Coh-Metrix on a corpus of 37,520 texts. Their analysis established eight indices to account for 67.3% of the variance. These eight indices include narrativity, syntactic simplicity, word concreteness, referential cohesion, deep cohesion, verb cohesion, connectivity, and temporality. Among them, the first five accounted for 54% of the variability in the texts. They also make up the indices reported by the Coh-Metrix Text Easability Assessment. The indices are reported as percentiles and z-scores in the output and described, albeit briefly.

A narrative text is characterized by having a story, characters, events, places, and things familiar to the reader. In contrast to non-narrative texts, narrative texts are written on more familiar topics, using oral language and more familiar words and world knowledge.³³ The higher the grade level of the text, the lower the narrativity index will be.¹⁹ Syntactic Simplicity is, however, based on the assumption that that there are fewer words, and simpler and more familiar syntactic structures in a syntactically simple text.³³ According to Graesser et al.,¹³ Coh-Metrix measures the degree of difficulty of the syntactic structure of the text in three ways: the mean number of

1. Modifiers per noun phrase;
2. Higher level constituents per word and
3. Word classes that signal logical or analytical difficulty

As the third index, Word Concreteness assesses the proportion of content words in the text that are concrete and meaningful and evoke mental images, as opposed to abstract words that are difficult to visualize.³³ Referential Cohesion is high if words and ideas overlap across sentences and the whole text, that is, the ideas are more clearly connected to each other and the text will be more easily processed by the reader. As the last index, Deep Cohesion goes beyond words and accounts for ‘causal, intentional, and temporal connectives’ Graesser et al.,¹² which “help the reader form a deeper and more coherent understanding of the causal events, processes, and actions in the text.³³”

Among the formulae reviewed above the FRES and FKGLs are mostly employed to evaluate textbooks written in English as a first language, the CMES is, however, intended to predict the readability of texts written in English as a second language.^6,34 This study aims to find out whether the FRES and FKGLs can be used to evaluate the readability of textbooks written in English as a second and foreign language on the one hand and whether FRES and CMES predict readability of those textbooks differently as do the FKGL and CMES do.

Textbooks

The first textbook whose readability was explored in this study is “High-Intermediate 3” [henceforth HI3] designed by the Iran Language Institute (ILI), former Iran America Society. According to its official website, i.e., http://ili.ir/Default.aspx?tabid=113, the ILI has 200 centers in 73 cities and offers English, French, German, Arabic, and Spanish to around 240000 learners. The HI3 is, therefore, read by a fairly large number of students who are learning English as a foreign language (EFL) in the ILI. It consists of ten units each of which consists of one reading comprehension passage as shown in Table 1. As can be seen, the Flesch-Kincaid Index Grade Level (FKGL) determined by Word ranges from grade 4 (units 4 and 10) to grade 9 (unit 5), indicating that the designers of HI3 have not paid any attention to the readability of its constituting reading passages. They deal with topics as diverse as “Eureka”, “What type are you?” and “Intuition”.

The difficulty of “New Headway Upper-Intermediate” Soars¹ [Henceforth NHUI], was explored as the second textbook. Although it is written for learners of English as a second language (ESL), it is taught at various institutes other than the ILI in Iran. It consists of 12 units as shown in Table 2. As can be seen, Soars and Soars have used neither FRES nor FKGL to choose or compose their 12 reading passages because their difficulty level varies from American grade 5 (units 3 and 12) to grade 12 (unit 4).

Procedures

Before analyzing and comparing the reading passages comprising HI3 and NHUI, they were first typed in Microsoft Office Word and were saved in .txt format as required by Coh-Metrix. Also, since Coh-Metrix measures number of paragraphs and the sentences per paragraph, typed texts were screened again in order to ensure that the typed sentences and paragraphs were exactly the same as the printed textbooks. Next, texts were submitted to Coh-Metrix website available at tea.cohmetrix.com and tool.cohmetrix.com for analysis.

Data analysis

The set of descriptive information obtained as part of Coh-Metrix output was first scrutinized to obtain a general overview of the passages comprising HI3 and NHUI presenting eight components in the form of z-scores, i.e., narrativity (NAR), syntactic simplicity (SS), word concreteness (WC), referential cohesion (RC), deep cohesion (DC), verb cohesion (VC), connectivity (CON), and temporality (TEM) and a single Coh-Metrix Easability Score (CMES), sets of Pearson correlations were computed to find out whether there was any significant relationship between the FRES and CMES as well as its components. These statistical analyses were conducted via IBM SPSS Statistics 23 to answer the following research questions.

1. Do CMES and its NAR, SS, WC, RC, DC, VC, CON and TEM components correlate significantly with the FRES scores obtained on passages comprising HI3?
2. Do CMES and its NAR, SS, WC, RC, DC, VC, CON and TEM components correlate significantly with the FKGLs obtained on passages comprising HI3?
3. Do CMES and its NAR, SS, WC, RC, DC, VC, CON and TEM components correlate significantly with the FRES scores obtained on passages comprising NHUI?
4. Do CMES and its NAR, SS, WC, RC, DC, VC, CON and TEM components correlate significantly with the FKGLs obtained on passages comprising NHUI?

Table 3 presents the descriptive statistics provided in the output of Coh-Metrix are mean sentence length, and mean word length. As can be seen, the mean number of words per sentence in the HI3 and NHUI are almost similar, i.e., 13.5 and 14.8, respectively. However, the former has a smaller SD value (7.3) compared to the latter (9.2), indicating that the length of sentences in NHUI passages have a wider range of dispersion around mean. The mean and SD of the syllables per word for the HI3 were 1.433 and 0.788 while those of NHUI were 1.443 and 0.761, respectively, indicating that neither HI3 nor NHUI contains longer and less frequent words to make it more difficult to read. This is because longer words tend to be less frequent in languages, and rarer words, in turn, tend to be more difficult.³⁵

Table 4 presents the second set of indices provided in the output of Coh-Metrix web tool as the Text Easability Principal Component Scores were checked for the passages comprising HI3. As can be seen, nine components provide readers with various types of information ranging from narrativity to temporality, entailing a relatively technical knowledge on the part of their users. Narrativity indices, for example, range from 41 (unit 8) to 98 (unit 9), with an average of 69. McNamara et al.,¹⁹ for example, claimed that the higher the grade level of the text, the lower the narrativity index will be. Higher narrativity is also associated with greater ease of comprehension.¹³

Table 5 presents the scores obtained on the componential analysis of 12 units comprising NHUI. As can be seen, the narrativity values of units vary from 25 (unit 2) to 97 (Unit 7) with an average of 55, indicating that they were difficult to read. Regarding other components McNamara et al.¹⁹ believed that they provide teachers with a variety of indices to consider when looking for a suitable text for their class and that these indices, altogether, provide a better picture of the essence of the text, compared to single measures of readability provided by traditional formulae. They will be discussed in the next section.

Table 6 presents the correlations between the FRES, CMES and its eight components obtained on HI3. As can be seen, the CMES does not correlate significantly with the FRES indicating that the readability score provided by Coh-Metrix has no relationship with the FRES. Among the components, however, only the NAR component correlates significantly with the FRES (r = .79, p<.01) and thus answers the first research question partially and positively. This result shows that FRES not only explains 62% of variance in narrativity as the first and most important component of Coh-Metrix but also highlights the fact that the more narrative EFL textbooks such as HI3 are, the more acceptable estimating their readability via the FRES becomes.

Table 7 presents the correlations between the FKGL, CMES and its eight components obtained on HI3. As can be seen, similar to FRES, FKGL does not correlate significantly with the CMES, indicating that as Coh-Metrix does not validate assigning American grade levels to the reading passages in HI3 as determined by the FKGL. However, the significant but negative correlation between the NAR and FKGL (r = -.70, p<.05) provides a positive answer to the second research question and shows that as a construct Coh-Matrix is not independent from FKGL because its first narrativity component does have significant relationship with the FKGL and explains 49% of its variance.

Table 8 presents the correlations between the FRES, CMES and its eight components obtained on NHUI. As can be seen, the CMES correlates significantly with the FRES (r = .87, p<.01). The FRES also correlates significantly with NAR (r = .92, p<.01), RC (r = .65, p<.05) and TEM (r = .60, p<.05) components of CMES and thus partially answers the third question positively. The highly significant relationship between the CMES and FRES does in fact show that the two scores provide essentially the same index of readability for the reading passages comprising the NHUI. The results also show that the FRES is empirically more related to the CMES than its SS, DC, VC, CON and TEM components are because they do not correlate significantly with the CMES as a theoretical construct.

Table 9 presents the correlations between the FKGL, CMES and its eight components obtained on NHUI. As can be seen, the FKGL correlates significantly but negatively with the CMES (r = -.85, p<.01). It also correlates significantly with three components of the CMES, i.e., NAR (r = -.87, p<.01), RC (r = -.60, p<.05), and TEM (r = -.83, p<.05). These results partially answer the fourth question and show that the American grade levels identified by the FKGL have strong relationships not only with the CMES but also with its NAR, RC and TEM components. Interestingly enough, the relationship between the NAR component of CMES and the CMES itself (r = -.86, p<.01) is even slightly stronger than the relationship between the CMES and FKGL (r = -.85, p<.01), indicating that the CMES as well as its narrativity component can easily be predicted by the FKGL.

The two textbooks HI3 and NHUI are both written and taught at the same level of English language proficiency, i.e., upper intermediate, in Iran. There is, however, no objective definition of the concept represented by the modifiers “upper intermediate”, i.e., schema, so that the textbooks could be evaluated on that basis. Council of Europe,³⁶ for example, divides ESL learners into basic, independent and proficient users. The independent or intermediate users are further divided into B1 and B2 levels. The reading comprehension of upper intermediate or B2 learners is defined as the ability to “obtain information, ideas and opinions from highly specialized sources within his/her field” (p. 70). The schemata represented by the words “information”, ‘ideas”, and “opinions” are all subjective in that the council does not specify what they stand for.

The present researchers believe the subjectivity of the schema “upper intermediate” has brought about the differences in the readability ease scores computed on the reading passages of HI3 and NHUI by the FRES and CMES. (The American grade levels identified by the FKGL and their relationships with the CMES and its components will not be discussed in this section for two reasons. First, they are irrelevant within the context of language proficiency where the students of a specific grade, say, grade three in senior high schools, may differ significantly from each other in terms of their proficiency. Secondly, as the results of this study show the FKGL correlates significantly though negatively with the CMES and some of its components.). Unfortunately, the anonymous designers of the HI3 provide their readers with no description or explanation of what the schema “High Intermediate 3” offered as the title of the textbook means to them because it is designed and taught within an EFL context, i.e., Iran. Soars et al.,³⁷ however, approach the synonym of the schema, i.e., Upper Intermediate, from two perspectives. They make a supposition on the part of English language learners as their readers and claim.

They have successfully achieved a certain amount. They have been introduced to a significant selection of the English language, grammatically, lexically, functionally and situationally, and possess sufficient language proficiency to be able to express themselves in a variety of social context. They can interact with authentic material as long as it is not of too specialized a nature, and can begin to feel ‘at home’ with English. It is certainly possible to engage them in conversation on all sorts of subjects without the interaction being too labored (p. 4).

It is not, for example, known what Soars et al.³⁷ mean by “certain amount”, “a significant selection”, “sufficient language proficiency”, “authentic material” “feel ‘at home’” and “too labored” because they are not students themselves. They have though tried to overcome the subjectivity of these terms through employing technical terms such as “grammar” and “function” which reflect two opposing approaches in language teaching, i.e., grammar translation and communicative language teaching. Their adoption of teachers’ point of view, however, presents their view of “upper intermediate” quite objectively.

Teachers might encounter a degree of frustration from their students – We do already Present Perfect many times before why we do again [italics in the original]? This (not invented) comment summarizes the upper-intermediate level [italics added]. The students have covered a lot in their four or five years of English, but very little comes out of their mouths or pens that doesn’t containing [sic] howling mistakes of some kind – tense, word order, wrong word, pronunciation, spelling, or whatever (p. 4).

In the quotation above there are words which represent the schema of “speaking”, i.e., “coming out of their mouths”, and “familiarity”, i.e., what is “covered a lot”. These two schemata are the key concepts upon which the NAR component of CMES is also based, i.e., “telling a story” and being “familiar to the reader”. In defining the component McNamara et al³³ announced that “Narrative text tells a story, with characters, events, places, and things that are familiar to the reader. Narrative is closely affiliated with everyday, oral conversation” (p. 85). They further claim that “[t]his robust component is highly affiliated with word familiarity, world knowledge, and oral language” (p. 85).

McNamara et al.³³ are, however, vague in terms of relating narrativity to “oral language” (p. 85) because there is no objective way of quantifying “oral language”. However, the variables upon which they have developed their NAR component show that narrativity basically consists of two variables, i.e., 1) the words comprising sentences and 2) the number of sentences constituting a text. These two variables do in fact comprise the FRES as well. For this very reason the NAR shows very strong relationship with the FRES (r = .92, p<.01) and explains 85 percent of its variance obtained on the reading passages of NHUI. The results of this study also show that since the correlation coefficient between NAR and CMES is higher than .80, i.e., .87, they can replace each other because they are empirically measuring the same construct.³⁸

The adoption of words and sentences as the main variables upon which the FRES and CMES are designed on the one hand, and selection of the 300 texts taken from the newspapers of two native English-speaking countries, i.e., Canada and New Zealand (100 texts each), and four non-native English speaking, i.e., Bulgaria, the Czech Republic, Portugal, and Romania (50 texts each) as the corpus of the CMES on the other, have brought about the strong relationship between the FRES and CMES as regards the passages comprising the NHUI. Since English is the L2 of the four countries whose newspaper articles have been used as the corpus of Coh-Metrix, then it can be argued that CMES has no empirical superiority over the FRES in determining the readability of L2 texts as claimed by the designers of Coh-Metrix.

Part of Soars¹ reading passage presented in the first unit of NHUI, “A home from home- two people describe their experiences of living abroad”, for example, is taken from a BBC News article, i.e., “Expat e-mail: Chile”.³⁹ It deals with a British man who applies for a job in Chile successfully and narrates his experiences working there. The HI3, however, contains descriptive reading passages which deal with educational topics. "Eureka!" (n.d.), for example, is the title of the passage given in unit 1. It is posted in a weblog whose author is not known. The very difference in employing the English language to present educational topics rather than cultural issues brings about no significant relationship between the Easability scores computed by the FRES and CMES on the reading passage of HI3.

However, when the narratitivity component of Coh-Metrics is taken into account as the main aspect of readability, both FRES and the NAR provide valid indices of readability for EFL textbooks such as the HI3 because they correlate significantly with each other (r = .79, p<.01). In other words, the FRES proves to be a concurrently valid measure of not only ESL but also EFL textbooks. According to the microstructural approach of schema theory (MICAST), both the FRES and CMES provide almost the same indices of readability because they are based on word tokens rather than word types. Adelpour’s⁴⁰ analysis of nine reading passages of “English Book 1” Birjandi et al.⁴¹ taught at grade one senior high schools (G1SHS), for example, show that the word types “the” and “he” have the highest tokens because they have been used 146 and 54 times, respectively.

Following traditional grammar and MICAST, Khodadady⁴² categorized the words constituting 22 news articles into syntactic words, i.e., “closed-class items” Quirk et al.⁴³ and semantic words, i.e., “open-class items”.⁴³ He also established a third category and named it parasyntactic words such as adverbs of frequency which are mainly syntactic because they are few in type and depend on semantic schemata to have meaning. Some parasyntactic words such as numerals are, nonetheless, many in type as semantic schemata such as nouns are. Similar to syntactic words, all parasyntactic words must, however, attach themselves to semantic words to have meaning. Khodadady⁴² argued that basing EFL teaching on tokens as the FRES and CMES do is misguided because they show that the syntactic words form almost the same percentage of texts (44%) as semantic words (42%) do.

In addition to being based on word tokens of specific reading passages, the FRES and CMES address individual passages and do not specify what word types constitute a specific textbook consisting of a number of passages and thus fail to address readability in terms of schema theory. By resorting to the MICAST, Khodadady et al.,⁴⁴ for example, argued that each word used in an authentic text represents a particular concept, i.e., schema, not only by itself but also in combination with other words constituting the text. This means that the more a specific semantic schema is used within a textbook, the more contexts the readers will have to relate it to other semantic schemata and thus enrich its conceptual scope. Unfortunately, neither the HI3 nor NHUI provide any clues as to what schemata their higher intermediate course subscribe to. Neither do the FRES and CMES indicate what words and sentences are specific to the HI3 and NHUI and distinguish them from other textbooks written from other levels of language proficiency.

In contrast to the FRES and CMES, the MICAST does, however, show that EFL textbooks written for students at Junior and senior high schools in Iran do differ from each other in terms of their schema types. Khodadady et al.⁴⁴ findings, for example, show that 57, 17 and 26 percent of word types comprising the EFL textbooks taught to junior high school students are semantic, syntactic and parasyntactic in domain, respectively. Adelpour’s⁴⁰ analysis of the textbook taught to grade one senior high school students, however, showed that the percentages were 67, 18 and 13, respectively, indicating that the semantic word types have increased 10% from junior to senior high schools. Future research must show whether similar changes can be found in EFL textbooks written for elementary, intermediate and advanced levels of EFL proficiency.

The concept of readability has been defined psychometrically in terms of difficulty faced by native speakers as they read various texts taught at elementary, secondary and tertiary levels of education in America. The main features of these texts, i.e., word and sentence length, which were captured by Flesch (1948) have prevailed even the CMES developed for ESL texts whose readers are not native speakers of English. The results of the present study show that the CMES and FRES hold very strong relationship with each other and can thus be utilized interchangeably.

Although McNamara et al.²⁹ claimed that CMES has superiority over other indices such as the FRES because it provides measures of narrativity, syntactic simplicity, word concreteness, referential cohesion, deep cohesion, verb cohesion, connectivity, and temporality, only one of these components is of empirical importance, i.e., narrativity. It not only correlates very highly with the FRES computed on the ESL texts but also can replace the CMES itself because it explains 85 percent of its variance. This finding questions the construct validity of ESL texts because they are based on the difficulties faced not by ESL learners but by native English speakers.

In contrast to ESL textbooks, EFL materials seem relatively independent from difficulties faced by native English speakers because the CMES and FRES computed on these texts do not relate significantly to each other. The narrativity component of CMES though correlates significantly with the FRES. The degree of relationship, however, does not make them interchangeable, indicating that EFL textbooks bring up topics which are fairly independent of what designers of readability indices impose on the language as a linguistic norm. Future research projects are though required to address the issue from other perspectives such as micro structural approach to schema theory.

None.

The author declares no conflict of interest.

1. Soars J, Soars J. New Headway upper–intermediate student’s book. 3rd ed. UK: OUP, Oxford; 2008.
2. Hewitt M, Homan S. Readability: A review of past research with a look at a new beginning. Journal of reading education. 1991;16(3):6–18.
3. Homan S, Hewitt M, Linder J. The development and validation of a formula for measuring single–sentence test item readability. Journal of Educational Measurement. 1994;31(4):349–358.
4. Brown JD, Janssen G, Trace J, et al. A Preliminary Study of Cloze Procedure as a Tool for Estimating English Readability for Russian Students. Second Language Studies. 2012;31(1):1–22.
5. Danielson KE. Readability Formulas: A Necessary Evil? Reading Horizons. 1987;27(3):178–188.
6. Crossley SA, Allen DB, McNamara DS. Text Readability and Intuitive Simplification: A Comparison of Readability Formulas. Reading in a Foreign Language. 2011;23(1):84–101.
7. Crossley SA, Dufty DF, McCarthy PM, et al. Toward a new readability: A mixed model approach. In: McNamara DS, Trafton G, editors. Proceedings of the 29th annual conference of the Cognitive Science Society. Cognitive Science Society, Austin, Texas, USA; 2007. p. 197–202.
8. Ozasa T, Weir GRS, Fukui M. A software application for assessing readability in the Japanese EFL context. Themes in Science and Technology Education. 2010;3(1):31–44.
9. Gilliam B, Peña SC, Mountain L. The Fry graph applied to Spanish readability. The Reading Teacher. 1980;33(4):426–430.
10. Brown JD. An EFL Readability Index. JALT Journal. 1998;20(2):7–36.
11. Crawford AN. A Spanish Language Fry–Type Readability Procedure: Elementary Level. Bilingual Education Paper Series. 1984;7(3):1– 20.
12. Graesser AC, McNamara DS, Louwerse MM, et al. Coh–Metrix: Analysis of text on cohesion and language. Behavior Research Methods, Instruments & Computers. 2004;36(2):193–202.
13. Graesser AC, McNamara DS, Kulikowich J. Coh–Metrix: Providing multilevel analyses of text characteristics. Educational Researcher. 2011;40(5):223–234.
14. Elfenbein A. Comments on Graesser, McNamara, and Kulikowich Research in Text and the Uses of Coh–Metrix. Educational Researcher. 2011;40(5):246–248.
15. Iran Language Institute. The ILI English series: High–Intermediate 3. Iran: Offset Printing House; 2006.
16. Dale E, Chall JS. The concept of readability. Elementary English. 1949;26(1):19–26.
17. DuBay WH. Smart language: Readers, readability, and the grading of text. Impact Information, Costa Mesa, CA, USA; 2007. 157 p.
18. Palmer HE. The principles of language study. UK: Harrup; 1921.
19. McNamara DS, Graesser AC, Cai Z, et al. Coh–Metrix Easability Components: Aligning Text Difficulty with Theories of Text Comprehension. USA: AERA; 2011. p. 1–12.
20. Zamanian M, Heydari P. Readability of Texts: State of the Art. Theory and Practice in Language Studies. 2012;2(1):43–53.
21. Si L, Callan J. A statistical model for scientific readability. In Proceedings of the Tenth International Conference on Information and Knowledge Management, USA; 2001. p. 574–576.
22. Flesch R. A new readability yardstick. J Appl Psychol. 1948;32(3):221–233.
23. Kincaid JP, Fishburne RP, Rogers RL, et al. Derivation of new readability formulas (Automated Readability index, Fog Count and Flesch Reading Ease Formula) for Navy enlisted personnel. Research Branch, Naval Technical Training Command, Millington, USA; 1975.
24. Flesch R. The art of readable writing. USA: Harper & Row; 1949. 254 p.
25. Gunning R. The technique of clear writing. New York: McGraw–Hill; 1952.
26. Fry EA. Readability Formula That Saves Time. The Journal of Reading. 1968;11(7):513–516.
27. Nguyen LT, Henkin AB. A readability formula for Vietnamese. Journal of Reading. 1982;26(3):243–251.
28. Nguyen LT, Henkin AB. A Second Generation Readability Formula for Vietnamese. Journal of Reading. 1985;29(3):219–225.
29. Greenfield G. Classic readability formulas in an EFL context: Are they valid for Japanese speakers? Ed.D. dissertation. USA: University Microfilms, Temple University; 1999.
30. Greenfield J. Readability Formulas for EFL. JALT Journal. 2004;26(1):5–24.
31. Bormuth JR. Development of standards of readability: Toward a rational criterion of passage performance. USA: U.S. Department of Health, Education, & Welfare; 1971. 219 p.
32. Graesser AC, McNamara DS. Computational analyses of multilevel discourse comprehension. Top Cogn Sci. 2011;3(2):371–398.
33. McNamara DS, Graesser AC, McCarthy P, et al. Automated evaluation of text and discourse with Coh–Metrix. UK: Cambridge University Press; 2014.
34. Crossley SA, Greenfield J, McNamara DS. Assessing Text Readability Using Cognitively Based Indices. TESOL Quarterly. 2008;42(3):475–493.
35. Zipf GK. Human Behavior and the Principle of Least Effort. New York: Addison–Wesley; 1949.
36. The Common European Framework of Reference for Languages: Learning, teaching and assessment. France: CUP; 2001. 273 p.
37. Soars J, Soars J, Sayer M, et al. New Headway upper–intermediate teacher’s book. The new ed. UK: OUP, Oxford; 2005.
38. Hatch E, Lazaraton A. The research manual: Design and statistics for applied linguistics. USA: Heinle & Heinle Publishers; 1991. 628 p.
39. Walker Smith I. Expat e–mail: Chile. UK: BBC; 2003.
40. Adelpour Z. Textual analysis of the “English Book 1” taught in Iranian high schools: Schema–based approach (Unpublished master’s thesis). Iran: Ferdowsi University of Mashhad; 2015.
41. Birjandi P, Soheili A, Noroozi M, et al. English Book 1. Iran: Textbook Publishing Company of Iran; 2013.
42. Khodadady E. Schema–based textual analysis of domain–controlled authentic texts. Iranian Journal of Language Studies (IJLS). 2008;2(4):431–448.
43. Quirk R, Greenbaum S, Leech G, et al. A comprehensive grammar of the English language. UK: Longman; 1985. p. 122–136.
44. Khodadady E, Hesarzadeh R. The effect of schema–vs–translation–based teaching on learning English in high schools. Theory and Practice in Language Studies. 2014;4(1):143–154.