Abstract
In the Indian context, concerns have been raised for many years about the learning outcomes of primary school children. The complexity of the issue makes it difficult to advise stakeholders on what needs to be done to improve learning in primary schools in India. As it has been shown that low socio-economic status is one of the key factors that negatively affect learning outcomes, the focus of the Multilila project (‘Multilingualism and multiliteracy: Raising learning outcomes in challenging contexts in primary schools across India’) is on educational achievement among children of low socio-economic status. In following the development of language, literacy, maths and cognitive abilities of primary school children over two years we hope to throw new light on why multilingual children in India do not always experience the cognitive advantages associated with multilingualism in other contexts. This paper focuses on some of the methodological challenges faced by this project. After explaining the rationale for the study, we sketch the contribution this project can make to the discussion about cognitive advantages of bilingualism. We then focus on the Indian context before presenting the methodology of the project (design, participants, instruments and procedure). Finally, we summarize the key challenges for the project and possible solutions to those challenges, and present an outlook towards the future.
Rationale for the Multilila project
Primary education has the potential to empower children and to support them in becoming agents of their own learning (Alexander, 2001). However, important differences exist within countries and between countries in the degree of success achieved in delivering high-quality education. Understanding the causes of these differences is extremely challenging because learning outcomes depend on the interplay between social, economic, demographic, linguistic, cognitive and pedagogical variables, which are difficult to disentangle. In the Indian context, concerns have been raised for many years about the learning outcomes of primary school children (see Banerji et al., 2013). The complexity of the issues makes it difficult to advise governments, teachers or parents on what needs to be done to improve learning in primary schools in India. In a detailed study of the determinants of poor learning outcomes in India, based on Annual Status of Education Reports (ASERs) and detailed household questionnaires, Alcott and Rose (2017) demonstrate that low socio-economic status (SES) is one of the key factors that negatively affect learning outcomes. For this reason, the focus of the Multilila project (‘Multilingualism and multiliteracy: Raising learning outcomes in challenging contexts in primary schools across India’) is on educational achievement among children of low SES and on providing evidence-based advice to stakeholders on ways forward in tackling the issues faced by this group of learners. For this longitudinal project, led by Ianthi Tsimpli from Cambridge and funded by the UK’s Economic and Social Research Council (ESRC) and Department for International Development (DFID) under the Raising Learning Outcomes grant scheme, an interdisciplinary team of researchers from the UK and India 1 follow the development of language, literacy, maths and cognitive abilities among 1200 low-SES children from Grade 4 (8–9-year-olds) to Grade 5 (9–10-year-olds) in government schools. It is novel not only because it is a longitudinal project, but also because of the wide focus on a range of variables, the attention paid to contextual factors (e.g. the differences between slum/non-slum areas) and the fact that data are collected across three different locations (Delhi, Hyderabad and Patna).
In the project we aim to find out in particular why multilingual children in India do not always experience the cognitive advantages associated with bilingualism to the same extent as other children who grow up with more than one language in different contexts (for an overview see Bialystok, 2009). Studying children in India is particularly interesting for this discussion because large cohorts of children with comparable language repertoires are found in the same classroom, which means that it is possible to focus on larger sample sizes than has been done hitherto in studies of the bilingual advantage (Paap et al., 2016). Carrying out this study would be very difficult in Western Europe, where school populations are often extremely diverse with 10 to 15 different first languages being spoken in the same classroom by children from very different cultural backgrounds (Trakulphadetkrai et al., 2017), which makes it impossible to find large relatively homogeneous groups of bilinguals in one school. Finally, as India has extensive experience with multilingual education (Mohanty et al., 2009; Panda et al., 2011), evidence of good practice in supporting multilingual learners can be shared with stakeholders in the UK, where multilingualism in schools has increased strongly since 2004 (National Association for Language Development in the Curriculum, 2014) and policy makers are struggling to develop appropriate policies to support the many learners of English as an Additional Language (EAL) in primary and secondary schools (Hutchinson, 2018).
We believe this interdisciplinary collaboration between researchers from education, linguistics, psychology and the social sciences based in India and the UK will facilitate a more in-depth understanding of children’s learning outcomes and cognitive skills than would have been possible in a monodisciplinary project. The focus of the current article is in particular on the methodological challenges faced by this project. We will first situate the project in the discussion on the cognitive advantages of bilingualism, before describing the Indian context in more detail. We then present the methods of the project (design, participants, instruments and procedure). Finally, the discussion reflects on the key challenges for the project and on possible solutions for these, and presents an outlook towards the future.
The contribution of the Multilila project to the debate about the cognitive advantages of bilingualism
There is a considerable body of research evidence showing that bilinguals or multilinguals outperform monolinguals in tasks that require high levels of cognitive control (for an overview see Bialystok, 2009). This difference is attributed to the fact that bilinguals constantly need to monitor which language they speak to different people and switch back and forth as the situation requires it. Advantages for bilinguals have been found in, for example, working memory (Morales et al., 2013), inhibition (Bialystok et al., 2004, 2006, 2008, 2012), metacognitive skills, for example cognitive flexibility (Bialystok and Viswanathan, 2009), creativity (Kharkhurin, 2012, for adults; Leikin, 2013, for children), inferential skills in oral narrative comprehension (Tsimpli et al., 2016) and analytic thought processes (Cummins, 2000). However, some of these findings are not always replicated (Colzato et al., 2008; Costa et al., 2009), raising the possibility that other factors, for example degree of proficiency in each language and socio-economic factors, might play an equally important role. Literacy levels have also been shown to interact with cognitive efficiency and speed of processing (Salthouse, 1996; Ziegler and Goswami, 2005), crystallized intelligence (Stanovich et al., 1995) and fluid intelligence, which is fundamental in problem-solving tasks (Kaufman et al., 2009; Kosmidis et al., 2011) as well as working memory (Andreou et al., in press).
As the bilingual advantage is only visible under some experimental conditions (see Costa et al., 2009), some observers have started questioning the existence of the bilingual advantage itself (Paap et al., 2016). It is difficult to obtain further insights into the bilingual advantage from studies based on cross-sectional designs for which monolingual and bilingual groups have to be matched on any variables that might affect executive functions. Clearly, this is virtually impossible because of the large number of factors that play a role here. Paap et al. (2016) therefore suggest longitudinal studies are needed to address this issue. In a longitudinal design, the comparability issue is avoided because bilinguals are their own controls: the main focus is on within-group developments over time. The Multilila project was set up to fill the gap in our knowledge about the development of cognitive skills over time and the relationship between these skills and literacy/language skills in challenging circumstances. It therefore has the potential to provide further insights into the specific circumstances under which any cognitive benefits of bilingualism can be found. As executive functions and other variables are measured at two time points, it also becomes possible to establish whether bilingualism at time one predicts executive functions at time two or vice versa.
The Indian context
As India does not take part in international tests aimed at ranking countries in terms of their educational performance, such as the Programme for International Student Assessment (PISA), which was introduced by the Organisation for Economic Co-operation and Development (OECD), 2 there is no recent information about India’s performance by comparison with most other countries (Banerji et al., 2013). The only comparisons available are those found in the BRICS Joint Statistical Publication (Ramos et al., 2017), which compares Brazil, the Russian Federation, India, China and South Africa. This report shows that India’s spending on education and educational performance falls below that of the other countries in the study. According to Parruck and Ghosh (2014), spending rose from 3.3% of Gross Domestic Product (GDP) in 2004–2005 to over 4% in 2011–2012, but in 2013 only 3.9% of GDP was spent on education, according to the World Bank, which is low by comparison with countries in Europe (e.g. 5.6% for the UK and 4.9% for Germany) and by comparison with countries participating in the BRICS study. Since then spending appears to have dropped. A report from the Centre for Budget and Governance Accountability shows that public spending on education has been less than 3% of GDP since 2012–2013 (Kundu et al., 2016).
Further details regarding educational attainment can be obtained from ASERs. These reports are based on annual household surveys conducted by Pratham, a non-governmental organization, among 600,000 participants in every district in India. ASERs consistently reveal low levels of learning outcomes in reading and maths and even report a downward trend between 2010 and 2014 (Banerji and Chavan, 2016; Pratham, 2017). Fortunately, the 2016 data for the whole of India show a slight increase in the percentage of children in Grade 3 who can read a Grade 1 level text (up from 40.2% in 2014 to 42.5% in 2016). A small increase was also observed for Grade 3 children being able to read a Grade 2 text (up from 23.6% in 2014 to 25.2% in 2016). While there is a gap between government schools and private schools, this appears to be narrowing in the lower grades. At the higher levels of primary school, by contrast, the gap does not appear to be narrowing. Important differences exist at regional and local levels, but overall the situation continues to give cause for concern.
As shown by Alcott and Rose (2017), a key factor that affects learning outcomes is social class. Children from low SES are more likely to underperform than children from more affluent middle-class families. Although middle-class children are more likely to attend private schools, as pointed out by the ASER 2016 report (Pratham, 2017), the differences between government schools and private schools cannot necessarily be attributed to differences in the effectiveness of these two school types because they might also be due to the interplay of a wide range of other variables, including SES, that affect learning.
Another important aspect of the Indian context is the fact that India is one of the most multilingual nations in the world. It is well known that giving precise numbers of languages is difficult for any country, because of the complexities involved in distinguishing language varieties (dialects) from each other and determining which ones count as ‘different languages’. According to the People’s Linguistic Survey of India, launched by Devy in 2010, there are 780 different languages in India, many of which are endangered (Devy, 2018). 3 Most recently, Ethnologue suggests the number of different languages is lower, and stands at 462 (Simons and Fennig, 2018).
Defining an appropriate language policy for education in such a linguistically diverse country is extremely challenging. The underlying principle of educational policy in India is the three-language formula, originally formulated in 1957, which postulates that all children should be taught through the medium of a regional language or mother tongue, to which an additional modern Indian language (e.g. Hindi) and English can be added as curricular subjects. The importance of the home language(s) or mother tongue(s) for children’s education is also emphasized in the National Curriculum Framework, which defines these as broadly as ‘the languages of the home, larger kinship group, street and neighbourhood, i.e. languages(s) that a child acquires naturally from her/his home and societal environment’ (National Council of Educational Research and Training, 2005: 36).
There are, however, concerns about the implementation of the three-language formula in schools (see Mohanty, 2006, 2008; Mohanty et al., 2010; Panda and Mohanty, 2015). Among the wider public, English is seen as a language of power and a gateway to improving one’s socio-economic position. For this reason, there is considerable parental pressure to introduce English as early as Grade 3 (or even at Grade 1), and to use English as the medium of instruction (EMI), particularly in private schools (Annamalai, 2013). The situation resembles that in many other countries in the world where EMI is promoted: in many cases levels of English remain low because the teachers’ own levels of English are limited, and appropriate resources are not available (Dearden, 2014; Erling et al., 2017). This means that some schools are English medium in name only, and actual teaching takes place in the regional or local languages (Annamalai, 2004; Mohanty et al., 2010). In these classes a considerable amount of code-switching or translanguaging (García and Wei, 2014) is likely to take place. It is possible that these facilitate learning but there is little systematic research into the impact on educational outcomes of pedagogical practices that allow for the use of different languages belonging to the children’s repertoires (but see Anderson, 2017; Jindal, 2013).
Large class sizes, poor resources and teacher-centred pedagogies also have a negative impact on learning outcomes (Rao et al., 2013). The role of pedagogical traditions in India is further analysed in Brinkmann (2015), who notes that teaching is still very teacher-centred despite efforts on the part of teacher training to develop more child-centred pedagogies. The development of critical thinking is not prioritized either (Dyer and Choksi, 2002), and there is little room for creativity or expression of independent thought (Jambunathan, 2005). On the more positive side, impressive initiatives to improve basic literacy and arithmetic skills among primary school children in India are undertaken as part of Pratham’s large-scale ‘Read India’ initiative (Banerji and Chavan, 2016).
Another complicating factor in the study is the presence of over-age children in primary schools. In India, children attend primary school between the ages of six and 10 (Parruck and Ghosh, 2014). At the time of the first data collection round, children in Grade 4 would therefore be expected to be 8–9 years old and one year later, when they are in Grade 5, they would be 9–10 years old. However, the age ranges could be much wider because some children are enrolled in school at later ages. According to Bhattacharjea et al. (2013), only half of all children enrolled in Grade 4 in government schools were eight or nine years old in 2012 and in some Grade 4 primary school classes one can find children as old as 13. While over-age children underperform in many contexts, as shown by Alcott and Rose (2017), the presence of over-age children is not necessarily a negative factor in India as in rural contexts these children appear to be more likely to be learning than children who are in the expected age range.
Methods of the Multilila project
As explained in the introductory section ‘Rationale for the Multilila project’, the current paper focuses on the methodological challenges faced by the project. In this section, we first describe the overall design of the project and the sampling procedure for the informants who took part in our study, after which we present the instruments and their administration. In addition, we report on issues encountered during piloting among learners who were comparable to the ones who participated in the main study. Finally, we present changes to the tasks based on the pilots. These were implemented to ensure the instruments were appropriate for the participants in the study.
Research design and participants
In this longitudinal project, participant recruitment was based on four variables: (a) geographical region; (b) gender; (c) SES; and (d) medium of instruction. A total of 1200 multilingual primary school children attending government schools take part at two points in time: once when they are in Grade 4 and one year later when they are in Grade 5.
Approximately 400 children were recruited from each of three geographical areas: two key metropolitan areas, Delhi (in the state of Haryana, North India) and Hyderabad (in the state of Telangana, South India), and Patna, a city in a more remote, relatively deprived area of the country in the state of Bihar, East India. These three areas were chosen to reflect differences in urban facilities available in each: while Delhi, the capital city, and Hyderabad are among the six metropolitan Indian cities with better infrastructure and more urban facilities, Patna is a district that consists of a small municipal area surrounded by rural blocks. Thus, the education facilities that government schools would have in these three areas are likely to reflect differences in educational infrastructures.
The three geographical regions were also chosen based on the languages available in these regions, which are as follows: Bihar (Patna): Hindi, Urdu, Bengali, Oriya, Maithili, Santhali, Bhojpuri and English; Delhi: Hindi, Bhojpuri, Maithali, Telugu, Odiya, Bengali, Assamese, Haryanvi and English; Hyderabad: English, Hindi, Telugu and Urdu. Not all these languages are also languages of instruction. The official medium of instruction in government schools in India is the state (or regional) language (Erling et al., 2017). In schools in our study, Hindi was therefore the expected language of instruction in Bihar and Delhi, and Telugu in Hyderabad, although some schools in Delhi and Hyderabad have an English-medium section or are officially classified as English medium. In classrooms where the official language of instruction is English, other languages, such as the regional language or its dialects, are often used in addition to English to support children’s learning (Bhagat and Panda, 2018; Dearden, 2014). We therefore decided to analyse this factor further and capture the multilinguality of the classrooms through classroom observations. The observations will also provide further information about pedagogical approaches in language and mathematics in the three different contexts.
Children attending government schools, which attract relatively little funding, are often from low SES. These children are also likely to have very little support for education at home. Their exposure to print is limited or absent except in market places. To understand the impact of SES on academic achievement, half of the children who took part in the study came from non-slum areas and the other half from slum areas. We assumed that the children from slum areas would grow up under more deprived circumstances than those from non-slum areas and these challenges may lead children living in slum areas to lower attendance in school because they would have to earn their livelihood or earn to add to their family income. This may also impact on their time and ability to engage with homework. The comparison between urban and rural areas in Delhi and Hyderabad on one hand and non-remote rural areas in Bihar, on the other, was also expected to show similarities between the children in urban slums and the children in the rural areas. During the pilot-testing phase, however, it was observed that the urban children in Delhi and Hyderabad had better educational exposure and school infrastructure, and relatively good levels of academic proficiency in their home language and moderate proficiency in English when compared with children from rural areas in the outskirts of Patna. These children had very limited or no knowledge of English and their school skills were lower than those of children living in urban areas.
The key variables used to select informants are represented in Figure 1. In participating schools, data were collected from all children attending Grade 4, to ensure the data collected at the school would constitute a valid representation of the ability levels of the entire cohort. In all schools we tried to ensure half of the children who took part were girls and the other half boys.
Key variables in the selection of informants.
Tasks, questionnaires and observation tools used in the study
An overview of all tools that were used can be found in Table 1. These are based on a battery of tasks that are widely used in empirical studies of multilingualism, for example in Tsimpli’s large scale EU-funded THALES bilingualism project, which assessed cognitive and language abilities of over 700 children in five different countries, and Marinis’ ESRC and Netherlands Organisation for Scientific Research (NWO) projects investigating language development in bilingual children across three different countries (Andreou et al., in press; Andreou and Tsimpli, 2017; Chondrogianni et al., 2015; Marinis et al., 2017; Rothou and Tsimpli, 2017). The tasks included the Raven’s Progressive Matrices, a narrative retelling task, a semantic fluency task, literacy and numeracy tasks, a Flanker task, an Updating (2-back) task and a maths anxiety task. The data for the school languages (Hindi and Telugu) and for English were collected on different days to avoid transfer effects. In addition, we developed a project-specific questionnaire about the participants’ background, a questionnaire for head teachers and teachers and an observation tool that allowed us to obtain further information about the languages used in the classroom and the educational activities in which code-switching and translanguaging were mostly found.
Task protocol for year 1 (4th Standard).
The questionnaires and the observation tool were developed in discussion with team members and consultants from India to ensure that they were appropriate for the Indian context. Following two project consultation meetings in India in July 2016 and July 2017, and one round of piloting the tasks and tools, we realized that the Indian context presented a certain number of linguistic and cultural challenges that made it necessary to adjust the methodology to ensure the tools were suitable for the Indian socio-cultural multilingual educational context. The changes adopted included:
task characteristic modifications;
linguistic modifications; and
cultural modifications.
For each of the tasks and tools we also report on the critical issues or ‘task difficulty’ faced during administration in the pilot-testing phase. All tasks were administered by research assistants who grew up and lived in India, spoke Hindi or Telugu in addition to some other regional languages and English, and were familiar with the Indian contexts in which the data were collected.
Cognitive tasks
Cognitive abilities, such as memory and attention skills, are known to underpin learning outcomes in monolingual and multilingual learners. They serve as predictors of academic success and may be related to multilingualism in a number of ways. For instance, proficiency in two languages and frequency of use have been shown to correlate with measures of cognitive control (Costa et al., 2009; Christ et al., 2011). Although most of the relevant studies focus on adult bilinguals, the role of bilingual education in cognitive control is relatively under-researched (but see Andreou et al., in press, for effects of bilingual education on working memory).
In the study we included three cognitive tasks, which measured general intelligence, updating, or inhibitory control: (a) Raven’s Coloured Progressive Matrices (CPM) (Raven et al., 2008); (b) a 2-back task; (c) a Flanker task. Two other measures of working memory were initially in the task battery but were eventually excluded. A verbal measure that is often used to measure working memory, that is, the backward digit span, was excluded because it was not possible to match the syllable length of number names in Hindi, Telugu and English. Therefore, it was decided to use only non-verbal measures to measure working memory and executive functions. A visuospatial working memory task was also considered but it was not included in the test battery because it was not a pertinent factor in this study and testing time had to be reduced to two hours per child. Below is a description of the three tasks used.
Procedure
The children carried out the tasks on a one-to-one basis with a research assistant. Instructions for all three cognitive measures were initially presented on a computer screen in English; however, during the piloting phase of the tests in Delhi in July 2017 it became apparent that the children were having difficulties comprehending the instructions as displayed on the screen. To facilitate the children’s comprehension, the research assistants gave oral instructions to the children in Hindi or Telugu. For the main data collection phase all instructions on the computer screen were presented either in Hindi or Telugu and the oral instructions were also prepared in these languages. All research assistants used the same set of instructions to maintain reliability of task instructions.
The children approached all the cognitive tasks with great enthusiasm because they associated laptops with games and assumed that they would play games on the laptop. The instructions for the Raven’s task were understood by most of the children and they were able to grasp the gist of the game fairly quickly. Research assistants helped the children up until the fifth slide wherever it was required.
The 2-back task involved multi-step instructions. Due to the complexity of the task, the children were slightly intimidated. There were occasions when the research assistants had to repeat the instructions and the practice sessions with the children to ensure they understood the game and were able to complete it. This task was the most difficult in terms of explaining the process, and even if the participant(s) could understand the process, at a later stage after seeing two to three numerals some children did not know when to press the ‘J’ button. They would look at the respective research assistant for further clarification, who had to remind them what they had to do. This meant that the first and second attempts of the children had to be discarded. Therefore, the time allocated to finish this task was longer than had been anticipated.
The children found the Flanker task very easy to complete, as visuals supported the verbal instructions. The only issue with the task was that it took nearly nine minutes to complete. Some of the students found it boring to press the left and right buttons, and therefore towards the end of the task their concentration levels were lower than at the start.
Assessment of semantic fluency
Verbal fluency tasks constitute one of the commonly used neuropsychological measures of cognitive functioning in bilinguals (Bialystok et al., 2008; Gollan et al., 2002). They are quick and easy to administer and, additionally, the tasks provide information on the development of both executive word-retrieval skills and lexical-semantic networks during childhood (Sauzéon et al., 2004) and are sensitive to a broad variety of disorders (Gollan et al., 2002). The performance on verbal fluency tasks can also be used to assess cognitive function with age and to diagnose abnormal patterns of cognitive function (Bialystok et al., 2008).
Several studies have investigated the performance of monolingual and bilingual children on verbal fluency tasks. Kormi-Nouri et al. (2003, 2012), studying bilingual and monolingual children, reported that there was a slight bilingual advantage in the letter fluency task in Persian, while in the semantic fluency task, monolinguals outperformed both bilingual groups. Bialystok and Feng (2011) tested monolingual and bilingual six-year-old children on semantic fluency of animal words and reported a slight advantage for the monolinguals (who generated a mean of 11.32 words) compared with the bilinguals, who generated 10.61 words on average, a difference that was not significant. From this existing research, results have been variable and the bilingual effect on semantic fluency may depend on academic experience and language proficiency in the second language (Kormi-Nouri et al., 2012); hence semantic fluency was considered worth exploring in the current study.
However, we did not include a letter fluency task in our study because of the difficulties involved in creating equivalent tasks across English, Telugu and Hindi. While according to Rosselli et al. (2002) there are no significant differences in the frequency with which the letters F, A and S are used in English and Spanish at the beginning of words, and therefore scores on a letter fluency task are comparable across these languages, it is much more difficult to make comparisons between alphabetic languages such as English and alphasyllabary languages such as Hindi and Telugu. In alphasyllabary languages, consonant–vowel sequences are written as a unit, while in alphabetic writing systems the status of vowels is equal to that of consonants. It is likely that phoneme awareness and letter knowledge are different in speakers of languages with different scripts (Nag and Snowling, 2012). Therefore, creating letter fluency tasks that are comparable between alphabetic and non-alphabetic languages is virtually impossible.
Scores on semantic fluency tasks are more likely to be comparable across languages, even though they could reflect domain-specific differences in usage of the two languages as bilinguals often use different languages for different topics and activities (Grosjean, 1998). Differences in scores across languages may also be an indication of differences in vocabulary knowledge in each language, as suggested by Unsworth et al. (2011), who found mid-strength correlations between vocabulary tasks and semantic fluency scores. It is generally assumed that bilinguals have smaller vocabularies in their respective languages than monolinguals (Bialystok et al., 2010). Thus, the tasks are likely to provide important information about the informants’ cognitive as well as linguistic skills.
For the semantic fluency tasks, which were administered on a one-to-one basis, children were asked to name as many entities as they could within one minute, belonging to two semantic categories:
living entities (e.g. animals for the home language and vegetables for the school language); and
non-living entities (e.g. household items for home language and school objects for school language).
When the semantic fluency task was administered in English during the piloting stage, some children said that they did not know any English words belonging to the required category. Many children could find words in the school language but naming the living and non-living entities in their home language was more difficult, which may be due to domain-specific uses of the different languages. Sometimes loanwords from English were used in the answers. Whenever these were established loan words (e.g. computer, car, teacher), as listed in Svobodová (2006), they were included in the word count.
Assessment of mathematical ability
A variety of tasks were used to assess children’s numerical understanding and their critical analysis and problem-solving skills. The ASER numeracy test was chosen because it has been widely used in the Indian context and its inclusion would therefore facilitate comparison of our results with those published in the ASERs. Permission to use the test was obtained from the ASER team prior to data collection. One component of this test was a number recognition task. Learners were directed to read aloud the numbers correctly in the language they were comfortable with, typically the school language. In addition, there were four subtraction problems (two digits) and two division problems (three digits by one digit). 4 These tasks were at the complexity level of Grade 2 for rural learners (Pratham, 2017) and, as shown in our pilot study, suitable for measuring mathematical ability among Indian learners from disadvantaged backgrounds. A complicating factor in the administration of this task was that there are differences in the mathematics curriculum across states (e.g. in relation to the use of fractions and division), which will need to be taken into account in the data analysis.
As the ASER tasks tap basic arithmetic ability, which is relatively independent of language, and we were interested in finding out to what extent language knowledge affected children’s mathematical ability, we also needed to obtain information about the children’s ability to solve mathematical word problems. Verschaffel et al. (2000: ix) define word problems as ‘verbal descriptions of problem situations wherein one or more questions are raised the answer to which can be obtained by the application of mathematical operations to numerical data available in the problem statement’. An example of such problems is: Kerosene comes in 5-litre cans. Ashoka needs 17 litres of kerosene for the household. How many cans must he buy? (Correct answer: 4). This and other word problems were adapted from the 2011 version of the Trends in International Mathematics and Science Study for Grade 4, which had been administered in 63 different countries worldwide. Children in India were unlikely to have seen these as India does not take part in tests aimed at establishing global educational rankings. Some cultural adaptations were made to facilitate comprehension (e.g. children can be asked to buy kerosene for the household but are not normally helping to paint a house, so the original formulation was changed to correspond to children’s experiences in India). One of the word problems could not be included in the final version because the weighing scale in the problem did not look like traditional weighing machines used in India (tarazoo, a weighing balance with a fulcrum and weight estimated in iron bars as scales). English names were changed to Indian names and included both Muslims and Hindus, girls and boys.
A third task was included to measure children’s meta-mathematical ability, that is their skills in critically analysing mathematical problems that had been solved by another student incorrectly. The children were required to identify and explain errors made in computing addition, subtraction and multiplication, which requires children to do more than follow an algorithm in reaching the solution and reflect on mathematical logic involved in solving mathematical problems. The task was developed by Panda and had been used previously in a longitudinal study conducted in Odisha and Andhra Pradesh (Panda et al., 2011). As during the pilot-testing phase the children were found to struggle with verbalizing why the mistake had occurred, we replaced the open-ended version with a multiple-choice type answer with four options, three of which reflected three degrees of meta-mathematic ability and the fourth was a ‘don’t know’ option.
After completing the maths tasks, children were given Hopko et al.’s (2003) Abbreviated Math Anxiety Scale (AMAS), which had been adapted for British children between the ages of 8 and 13 by Carey et al. (2017). This two-dimensional scale consists of nine items which tap into Learning Maths Anxiety and Maths Evaluation Anxiety. Children give answers to the items on a five-point scale (with emojis representing smiling or sad faces to express (dis)agreement with the items). The AMAS was included to enable us to obtain a more in-depth understanding of the reasons why some children perform less well in maths. Administering this task was complex because emotions are not easily translatable across languages and cultures (Wierzbicka, 1999) and this turned out to be a complex issue for the concept of anxiety too, which has no translation equivalent in Indian languages (Ganesh Devy and Minati Panda, personal communication, 2017). The closest Hindi translation equivalents are /bʰəj/, /dər/, /gʰəbərɑhət/ etc., which mean fear rather than anxiety, /tʃɪnt̪ɑ/ (between worry and anxiety) or /fɪkər/ (between anxiety and care/caution). Another issue with the task was that children were not always aware that emojis can be used as a medium for conveying a response. As a result, even if children understood the question with the help of a research assistant, they could not always exactly pinpoint the emoji that would be the best fit for a response. This may be due to the fact that the concept of emojis is relatively new to the Indian context as compared with western countries, even though the telecom revolution has ensured access and availability of mobile phones in households from different socio-economic backgrounds.
The ASER arithmetic tasks were initially group-administered, but this took more class time than had been anticipated. As the speed with which children solved the tasks was not uniform, it was decided to administer the maths tasks on a one-to-one basis and outside class time. All the mathematics tasks (word problems and meta-maths) were translated into Hindi and Telugu. ASER arithmetic tasks were already available in Hindi, Telugu and English. The version that matched the learners’ medium of instruction was used. In Patna, for instance, mathematics tasks were administered only in Hindi while in Hyderabad they were administered in English and/or Telugu and in Delhi in English and/or Hindi depending on the medium of instruction in each school. Care was taken to ensure the variety of Hindi/Telugu used in the task corresponded to the colloquial variety of the languages spoken by the children. The number recognition task was found to be the easiest among the basic numeracy tasks, but many learners were struggling with division and complained that they were not taught how to solve division problems with three digits. As expected the word problems were among the most challenging ones. Although the questions were posed in simple language either in L1 or in L2, many children found it very difficult to extract the arithmetic problem from the narrative. When children failed to understand the written instruction, they were given explanations by research assistants in Hindi or Telugu and the children relied heavily on these.
Assessment of literacy skills
The ASER literacy tasks were chosen to assess students’ levels of literacy in the medium of instruction in their school, that is either the regional language (Hindi or Telugu), or English. The test included tasks at different levels of complexity: naming letters, single word reading, reading of sentences and short passages. Only children who were able to complete the sentence-reading tasks were invited to read the short passage, and to answer two reading comprehension questions about the passage. As children were more familiar with the regional languages, it was noted that they felt more anxious whenever they were required to read in English. Although learners were given five minutes preparation time before they were expected to start reading the text aloud, many learners were unable to read the text properly. There were numerous pauses and hesitations in their speech. Some of the commonly mispronounced words included garden, alone, lonely, seed, beak and dropped. Children struggled with the comprehension of the story. They asked for a translation of the comprehension questions in Hindi or Telugu and gave the answers in those languages only. In some cases, learners chose to point to the passages in the test that contained the relevant information without articulating the answers.
Assessment of oral skills
As narrative skills have been closely linked to a child’s language and literacy development, we have included an instrument to measure these skills in the test battery. The Multilingual Assessment Instrument for Narratives (MAIN, Gagarina et al., 2012, 2016) was chosen for this project as MAIN was developed as an instrument that could be used to elicit narratives from children from diverse linguistic, socioeconomic and cultural backgrounds. Furthermore, MAIN was designed to enable researchers and clinicians to distinguish between bilingual children with and without language delays or impairments.
Details controlled for in the instrument include: (a) story characters: the number of protagonists, the timing of the introduction of new protagonists, their relative spatial position in the stimulus pictures and interaction with other elements in the picture (e.g. their size in relation to other objects), and the angle from which they were looking at the other protagonists; (b) background and foreground information: the protagonist’s actions defined the foreground in each story and the pictures were of similar cognitive complexity and visual density; and (c) content: comparable onset, development and conclusion of the storyline. MAIN includes four picture-strips corresponding to four different stories. The tool also includes a scoring protocol for scoring microstructure, such as narrative length and lexical diversity, morphosyntactic complexity and discourse cohesion (mean length of communication units and of the three longest communication units), as well as syntactic complexity (the number and ratio of verb-based clauses) and types of subordinate clauses. Macrostructure, that is, components of the story episodes that are essential for establishing coherence in temporal and causal relations, are also scored according to criteria set out in MAIN. In addition, macrostructure measures include scoring the production of internal state terms that include any linguistic expressions that refer to the story characters’ feelings, intentions, goals and reactions to unfolding events. Finally, a set of comprehension questions for each story are included to evaluate the child’s understanding of the story components and internal states of story characters, such as goals, intentions and reactions to episode and story outcomes. It would also be a holistic assessment of their linguistic abilities from lexis and syntax to discourse.
Along with the retelling task, a few comprehension questions were asked to assess knowledge of ‘text connecting links’ and ‘gap-filling’ (Chikalanga, 1992). The comprehension questions from MAIN (see baseline tasks) are to assess the children’s ability to reason about the physical and emotional cause–effect relationships, as well as the ability to recognize characters’ goals, the reasons for these goals and the reactions that follow the attempts to reach the goals. They are an example of a problem-solving task by answering narrative text-based comprehension questions (Gagarina et al., 2012).
We had originally planned to also include narrative telling alongside narrative retelling in the project. This task was dropped from the battery because it was felt that the cognitive–linguistic load to build content on their own would be challenging for children from low SES. So, learners could listen to the text in English but could respond in Hindi or Telugu. A record of their choice was included on the child’s audio file. Instances of code-switching/translanguaging were accepted as valid responses.
In schools where English was the official medium of instruction, the English version was used, but in schools where the medium of instruction was Hindi/Telugu, the audio input was in the same language. In Patna, for instance, all schools have Hindi as the medium of instruction and only the Hindi version of the task was administered. In some cases, the students from English-medium schools wanted to listen to the story once again or asked for a Hindi/Telugu version to be played. The story was replayed in some cases but since using the Hindi/Telugu version was not part of the instructions for administering the task in English-medium schools, some students may have lost out on the opportunity to understand and retell the story in a more structured and cohesive manner. Some learners did not have any prior exposure to narrating stories either by listening or from memory, which made this a challenging task for them. However, in this study the learners were given the choice to retell the narratives in Hindi or Telugu even if the official medium of instruction was English. This choice was given so that they could express their thoughts in a language they felt comfortable with. During administration it was noted that children generally preferred questions to be asked in Hindi or Telugu rather than in English.
Questionnaires and classroom observation tools
The questionnaires and the observation tools were adapted from existing questionnaires that had been used in the Indian context in other projects. These included the Household Questionnaire, developed by Pauline Rose and Nidhi Singal from the University of Cambridge as part of the Teaching Effectively All Children (TEACh) project. 5 We also used questionnaires for children, teachers and headteachers and a school observation tool from the Young Lives project in India (https://www.younglives.org.uk/content/india-school-survey).
The teacher questionnaire focused on teacher qualifications and their pedagogical practices as well as teachers’ perceptions of the role of the mother tongue in primary education, and good practice in teaching and assessment in English, regional languages, literacy and maths.
The aim of the head teacher questionnaire was to obtain more information about the following issues: (a) demographic details about the school population, such as attendance records and drop-out rates; (b) head teacher qualifications and his/her professional experience; (c) knowledge about the school curriculum and pedagogical practices of teachers; and (d) the head teacher’s perceptions about key issues in education (e.g. the role of the mother tongue, English-medium instruction, usefulness of the three-language policy in school, differences between boys and girls in educational achievement). As some head teachers suggested the teachers were a more reliable source of information about children’s attendance records, teachers were also consulted on this issue.
The child questionnaire was used to gather information about the children’s language knowledge and language use at home and at school, personal background information, and whether or not they received support for learning outside school. As most children were not aware of their date of birth and age, there was often a mismatch in the information given by the children and the school records. To resolve any discrepancy, research assistants compared all the information collected from the students with the school records. The information given in the school records was considered to be final. Obtaining information from the children about the languages they spoke was not easy because children often did not know the names of the languages they spoke (apart from Telugu, Hindi or English). For instance, some children referred to the local varieties as Jo gaon mein boli jati hai ‘one that is spoken in the village’, and others called their local varieties Bihari, Rajasthani or Pahari, which did not correspond to the official names of languages. Sometimes the labels used referred to a collection of languages from the areas where the children lived. In all cases the children’s labels for the languages were recorded. Some children reported receptive knowledge of some languages (e.g. ‘Nepali’ as spoken by a grandfather). Another child reported that he knew ‘Bihari’ but used this language only to his grandparents, as they did not understand any other language. The child reported not being allowed to speak ‘Bihari’ at home as the father discouraged its use because of the taboo associated with the language in urban cities like Delhi.
Many children found it difficult to recall when they had arrived in Delhi. The research assistants had to ask a range of questions to obtain an estimate of arrival dates. These included questions such as: Are you studying in the same school from class 1? If not, where were you studying earlier? This provided some information as to whether the child was already in Delhi at the age of six or not. Other questions that turned out to be difficult to answer were removed from the questionnaire. These included questions about video games, because children did not have access to computers; questions about emotions or children’s personal lives; or questions about time spent on different activities outside school. The children did not feel comfortable talking about these issues and found assessing the amount of time spent on different activities outside school very difficult.
A classroom observation tool was developed to obtain in-depth information about actual teaching practice, and in particular the use of languages in the classroom. This was particularly important for obtaining further insights into any discrepancies between the school’s official language of instruction and the actual languages spoken in class. In India, pedagogical practice in many schools is teacher- and textbook-centred. We therefore attempted to gauge to what extent learning was teacher-directed and/or student-led, using a ‘good practice’ table developed by the British Council India. In addition, for participating teachers, 30 minutes of their activities in a language classroom and in a maths classroom were observed as part of the project. Teachers’ and learners’ activities were coded (e.g. reading aloud, asking questions, problem-solving exercises) and languages spoken during the 30 minutes were coded at five-minute intervals, to enable the researchers to obtain detailed information about the languages used and any translanguaging practices (i.e. use of children’s L1s in the classroom).
Discussion and conclusion
As ASERs have shown for many years, the learning outcomes in reading and maths of children attending government primary schools continue to be very worrying, although there is no information about India’s performance on global indicators of educational achievement. Any efforts to improve the situation will need to be based on a thorough understanding of the causes for the low levels of achievement in schools, as the economic factors interact with a variety of social, linguistic, cognitive and pedagogical factors. The aim of the Multilila project is to obtain a deeper understanding of the interplay between these factors and how these affect children’s performance on a range of literacy, language, math and cognitive tasks. As most Indian children know more than one language, they might be expected to experience the same cognitive benefits that multilingual children have been found to have in other countries (Bialystok, 2009). Finding out to what extent this is the case, and under which circumstances any such advantages emerge, is one of the key aims of the Multilila project.
The project is likely to be able to contribute to the discussion about these issues because through its longitudinal design it avoids the difficulties involved in comparing monolinguals and multilinguals. Such comparisons are considered to be increasingly problematic, because language processing in multilinguals is fundamentally different from that in monolinguals (Grosjean, 1998). In addition, monolinguals’ language proficiency is not uniform, which makes it difficult to identify an appropriate monolingual baseline (Andringa, 2014). Because bilinguals are often compared to monolinguals in the bilinguals’ weaker language (Ortega, 2010), findings are likely to show that multilinguals are less proficient than monolinguals.
The Multilila project is original in that it has a wide focus on a range of outcome variables (maths, language, literacy and cognition) and data collection takes place in three different states in India. Carrying out such a longitudinal project is inevitably very challenging. The aim of the current article was to sketch the methodological challenges and the solutions adopted in the project. Information about contextual factors that affect learning is obtained through questionnaires for (head)teachers and children which provide additional background information. It is anticipated that these can provide an in-depth picture of children’s abilities, even though we are aware of the impossibility of including all variables that might affect learning. We are aware that for economically and socially underprivileged students who have less exposure to pedagogically driven everyday conversations, taking tests may be particularly challenging, but the research assistants could explain tasks in ways that the children were able to understand.
The tools were chosen and/or adapted in consultation between members of the research team and consultants in India. After piloting, linguistic and cultural adaptations were made to many tasks to ensure they would be suitable for the target groups in the Indian context. Tasks that involved the expression of emotions (e.g. anxiety in the mathematics anxiety task) proved to be challenging as such terms cannot be easily translated into other languages. Context-sensitive solutions were found for these issues in collaboration with experts from different fields.
Because pedagogical practices in India tend to rely on memorization, copying from the blackboard and rote learning, tasks which required children to work independently on a problem proved to be challenging. The problems faced in administering the word problems revealed lack of development of independent reading and comprehension skills among students. Discrepancies between the official medium of instruction in Delhi schools and actual language use in class meant that some schools turned out to be English medium in name only, which made the choice of languages for tasks a complex issue. In some schools pedagogical practices relied on switching between two different languages or between two varieties of the same language or translanguaging. An important question that will need to be addressed as part of future analyses of the classroom observations is how switching/translanguaging impacts on the development of reading and maths among the students. Further evidence about the actual practices can therefore provide important information for policy makers in India and the UK interested in raising learning outcomes among children from low SES in challenging circumstances.
Footnotes
Acknowledgements
We sincerely thank all the research assistants, participants and teachers for their diligent support and earnest endeavours throughout this project. We are very grateful to Dr Debanjan Chakrabarti, Prof. Ganesh Devy, Dr Dhir Jhingran, Prof. Vasanta Duggirala, Ms Amy Lightfoot, Prof. Rama Matthew, Prof. Ajiit Mohanty and Prof. Bapi Raju for their advice and support throughout this project.
Declaration of conflicting interests
The authors declared no potential conflicts of interest with respect to the research, authorship and/or publication of this article.
Funding
The author(s) disclosed receipt of the following financial support for the research, authorship, and/or publication of this article: This research was funded by DFID and the ESRC, grant number ESRC-DFID, grant ref: ES/N010345/1.