Pencils of Promise (PoP) has worked in Laos for ten years and implemented our Teacher Support (TS) program there for the first time in 2012. As the first country to receive our TS program, our intervention with teachers has seen many iterations over the years. There are unique challenges associated with our work in Laos and with our support of primary-level English teachers.
First and foremost, students are introduced to formal English instruction in school for the first time in the third grade and it is often the third or fourth language these children are exposed to and/or learning*. Secondly, English instruction is offered one day a week for one and a half hours. This places great importance on the ability of PoP-supported teachers to effectively deliver complex content on a foreign language (i.e., English) in such a limited amount of time. Our TS team continues to make progress with improving skills of teachers in the classroom and, as PoP prepares to expand to supporting both third and fourth grade teachers during the 2018–2019 school year, our Learning & Evaluation (L&E) team strives to assess our work in the most reliable and valid means possible to report on our impact.
Historically, we have relied on the Early Grade Reading Assessment (EGRA) to understand changes in student performance as a result of being taught by PoP-supported teachers. Through a critical reflection of our approach to evaluation and an internal assessment of the effectiveness of each tool, we identified a gap between the complexity of the EGRA and the simplicity of English education at the third grade in Laos. As Jonathan Tan, former PoP employee and current member of Graphite: Impact Council, wrote in May 2018 for Transparency Talk, our disconnect with EGRA in Laos led us to conclude that ASER served as a much better fit to the beginner level of English exposure among PoP students.
Moving to the ASER
The ASER (aser translates to impact in Hindustani), or originally called the Annual Status of Education Report, began as a national assessment in India used to determine basic English literacy levels among students and was designed to be administered at the community level and/or in the home (Figure 1). This form of citizen-led assessment expanded the coverage of literacy evaluation for students, especially out-of-school children, and empowered community members to participate in the process. Students assessed are assigned a “fluency” level based on their ability to identify letters, identify words, read simple sentences and read and comprehend a short paragraph. An important feature of the ASER is its ability to be manipulated based on the curriculum, skill level and context in which it’s being used.
After consult from Pratham and the ASER Centre, we set out to develop a version of the ASER that would be a reliable and valid measure of literacy levels among third and fourth-grade students in Laos. We developed a pilot testing plan that would enable us to assess random groups of third and fourth-grade students, analyze results, adapt our testing instrument and then test a different group of students. PoP’s L&E team in Laos organized and implemented three separate rounds of pilot testing over a three-month period (May — July 2018).
Methodology for Developing the PoP ASER
In our first version (Figure 2), we decided to omit the paragraph section, as students are not introduced to this level of reading at the third or fourth-grade level. We also wanted to learn if there was a difference between word type (i.e., phonetic vs. sight**) and letter case (i.e., upper vs. lower), so appropriate sections were included. Content was sourced primarily from the Lao curriculum, though a few sentences and words outside the curriculum were added to provide an increased level of difficulty.
Given the brevity of the assessment compared to the large amount covered in the English curriculum throughout the school year, sentences, words and letters have to be selected at random, which helps provide a representative sample of content and minimizes bias. For sentences and words, content was selected with the intent of representing lessons covered throughout the school year (i.e., items were selected from the beginning, middle and end of the curriculum). Letters were selected at random, with equal representation from three different tiers of frequency in the English language (Figure 3). We decided to include more content than outlined in the ASER example (Figure 1) to increase the validity (i.e., more targeted content leads to an increased representation of the curriculum) and still believe the assessment will not lead to test fatigue.
Pilot Testing, Round 1
Results from the first round of pilot testing revealed that students were unable to read any of the sentences and were only able to identify a couple of the words, with no difference between sight or phonetic. As for letters, students were more likely to identify uppercase letters, though not enough to be deemed “fluent” in the section (students averaged 3 out of 12 with one student (out of 47) being able to identify 6 out of 12).
These results were presented to the TS team in Laos, where Whitney Wilson, PoP’s Laos Education Training Consultant, pointed out that the lack of a student’s ability to read sentences, or even identify words, was predictable given that the curriculum focused more on communicating than reading. An English lesson in Laos consists of presentation of a letter, word association with the presented letter and then image association with the word. Short and simple sentences are used and presented during the image section of the lesson, though emphasis is placed on communicating what the image is. For example, the letter B would be followed by the word box, which would then be followed by an image of a box and the sentence “This is a box.” This highlights the focus on image association, a beginning step in communicating in a new language, rather than reinforcing reading.
This revelation encouraged us to think of a way our version of the ASER could align more with the English curriculum in Laos and avoid the overabundance of irrelevant data (e.g., zero scores on an assessment that do not change from pre to post testing periods). Rather, it is not advantageous to have an overly difficult assessment as it will become a poor predictor of the intervention (i.e., PoP TS). Sentences on Version 2 of the PoP ASER (Figure 4) were replaced with images, which the student would be asked to identify using the English word. Version 2 kept a word section, though it was condensed by combining sight and phonetic word types, given the lack of differences identified in the first version. Letters were again separated into upper and lowercase and selected at random using the same methodology.
Pilot Testing, Round 2
Of the 40 students assessed in the second round of piloting (i.e., Version 2), 21% to 28% were able to identify three of the nine images (elephant, box, and “five”). None of the students were able to identify hand, head (image 5), walk (image 7) and table, while a minimal number of students could identify monkey and green (image 6). The decrease in zero scores from sentences to images suggested that the image section is more aligned with the content taught in the classroom and a more relevant predictor for student performance. Students are more familiar with these images and identification is aligned with their curriculum. While word identification remained low (none of the student were able to identify any of the 12 words in Version 2), it is important to keep this section in the development of future versions as the introduction of words is part of the English curriculum in Laos and the development of word recognition among students is a goal of the PoP Teacher Support program.
Pilot Testing, Round 3
In the development of Version 3 (Figure 5), our team aimed to refine the images based on feedback from the test administrators. Students and assessors reported that the images of head, green and walk were confusing; therefore, these were replaced with sun, pen and wallet. Sun and pen are both images included within the Lao curriculum for English, so the inclusion of wallet was intended to create a challenge for students. Version 3 included the same list of words and letters from Version 2, which was not determined to be a limitation given a new random sample of students would be assessed.
49 3rd and 4th graders were included in the Version 3 PoP ASER pilot assessment. This group of students were a completely different sample from previous pilot rounds. Nearly half (49%) of students were able to identify the number “five”, while none of the students could identify wallet or hand. Other images ranged in totals of 12% to 35% of students assessed. Interestingly, recognition of table increased from 0% on Version 2 to 24% on Version 3, suggesting that this image successfully measures curriculum-relevant content and could remain on future versions. There was a minimal increase in word recognition from previous pilot rounds, with 3% of student being able to identify the word table. Zero scores across all students remained relatively high with no students being able to identify half of the 12 words. Letter recognition remain similar to previous versions. There is more consistency in recognition with uppercase letters, though z (43%) and o (29%) continued to be vastly more recognized than other letters.
After completion of three rounds of pilot testing with more than 100 students across five different schools, our team reflected on observations, methodologies and test scores to design a final 3rd and 4th-grade version of the PoP ASER. For each of the assessment sections, final versions were based on the following hypotheses:
- Letters: Random selection of 12 upper and lowercase letters in three different tiers of letter frequency (Figure 3) provides a large selection of letters, which are all covered in the 3rd and 4th grade English curricula. Results from piloted version does not guide the selection of letters, as random selection of the foundational component of a language (i.e., letters) yields a reliable and valid assessment of student ability.
- Words: Random selection of words from the beginning, middle and end of the 3rd and 4th-grade curricula will present a broad representation of words covered throughout the school year across various topics (e.g., numbers, anatomical terms, objects, etc.). Low recognition of words throughout all pilot test versions suggest there is no difference between sight and phonetic words (i.e., results from Version 1) and challenging words (i.e., words outside of the curriculum) should not be included in the assessment due to introductory level of students.
- Images: Content was selected at random throughout the 3rd and 4th-grade curricula in order to gain a broad representation of lessons presented throughout the school year. Results gathered during the pilot version were especially informative when finalizing images, as students would report that some images were too confusing. These would be replaced with less complex images, while still representing key pieces of the curricula (e.g., numbers, anatomical terms, objects, etc.). For example, in Version 3 of the assessment, the image of a wallet was very confusing for the students. This was replaced with a red square, as shapes and colors are key components of the 3rd-grade curriculum that were not yet covered on the ASER.
With these methodological considerations, a final version (Figure 6) of the 3rd grade ASER was developed, with changes from Version 3 being made based on feedback provided by students and observations from assessors. The same methods were used to develop a 4th-grade ASER (Figure 7). Piloting of a 4th grade version did not need to take place, as the methods used for content selection were 1) not specific to, or specialized for, the 3rd grade curriculum and 2) both the 3rd and 4th grade curricula are structured the same (e.g., use of “letter-word-image” as a method of instruction). Additionally, 4th graders were included in piloting of the 3rd-grade versions and did not show a significant difference in performance compared to the 3rd graders, which suggests that 4th graders should be assessed with simple content.
Next steps
Baseline data collection in the 3rd and 4th grades was completed in early-October 2018, with Endline scheduled for the close of the school year (May 2019). PoP’s L&E team successfully assessed 526 3rd graders and 525 4th graders, who will all be included in the Endline sample (April/May 2019). While we are pleased with the introduction of the ASER, we still see room for improvement in the assessment. First and foremost, improvements must be made in the generation of content for words and images. The approach taken in the first round of test development could be defined as inherently subjective and, therefore, a more valid and robust system for selection must be developed. Our L&E team in New York has already taken steps toward researching reliable and valid methods for content selection, as Julia Carvalho, Impact Manager of Research & Development, has conducted in-depth research on selecting content for PoP’s new modified-EGRA in Guatemala (future Transparency Talk post).
It should also be noted that the PoP ASER will only be administered in schools. As previously mentioned, the ASER was designed as a citizen-led assessment, which offers an increased opportunity in flexibility for testing environment. Given that PoP currently does not provide any support outside of the school environment, our use of the ASER will remain in the classroom.
We are excited to move forward with the ASER and believe it is a strong fit for our programming in Laos and, therefore, will reflect more useful information. Our literacy assessments at PoP (ASER and EGRA) are just one of several data points we use to inform and improve our work. Refining and improving each evaluation tool increases our efficiency and enables us to better serve students, teachers and communities impacted by PoP. If you have any questions about the development of the PoP ASER, please contact PoP’s L&E team directly at cstanfill@pencilsofpromise.org.
Footnotes:
*PoP communities in Laos are often ethnically Khmu or Hmong, which means children exposed to at least two languages (ethnic mother tongue and Lao (national language and language of instruction throughout school)) at a very young age.
**Phonetic or decodable words contain letters that can all be sounded out individually and the word will be read correctly. Sight words (also referred to as high-frequency sight words) contain letters that, if sounded out phonetically, will lead to the incorrect reading of the word. This includes silent letters or letter combinations that hold one phonetic sound.