Teaching Support (TS) programming at Pencils of Promise (PoP) started in Luang Prabang, Laos during the 2012–2013 school year. When PoP launched the TS program in Laos, Momosign — a language learning program designed to incorporate “visual-spatial qualities of sign-language communication” to teach English as a Foreign Language in primary school — was incorporated as a critical piece of curriculum delivery. In early 2017, PoP was interested in testing the effectiveness of Momosign outside of the context of the PoP-supported classroom. This would enable an external evaluation of the effectiveness of Momosign, while controlling for the influence of PoP’s TS program.

PoP and the Education for Development Fund (EDF) in Laos partnered to make this external evaluation a reality. EDF works in several provinces throughout Laos and, among several other programs, trains teachers in the Momosign approach to teach English. In their model for Momosign delivery, EDF provides:

• One Momosign book for each student
• One workshop for each teacher
• One computer for each school

EDF’s training workshop for the teachers takes place over a four-day period and includes topics such as classroom observation and use of lesson-related technology, though the majority of the training period focused on the seven-step Momosign lesson plan and practice of delivery of the lesson plan.

Approach

The Early Grade Reading Assessment (EGRA) is a toolkit developed to measure the skills necessary to acquiring literacy. The specific skills tested are the reading and writing of letters, words and simple sentences. In Laos, PoP uses a modified EGRA-based assessment to evaluate the effect of our interventions on student performance. Figure 1 below provides a list of names and descriptions for each EGRA subtask.

Subtask Description

• S1a: Letter Identification — Student is asked to say the names of 10 provided letters.
• S1b: Letter Writing — Student is asked to write 10 letters read aloud by the enumerator.
• S1c: Letter Cases — Student is provided a prompt with 5 upper- or lowercase letters, and is asked to provide the corresponding lower- or uppercase version.
• S2: Letter Sounds — Student is asked to say the sounds of 10 provided letters.
• S3a: Reading Simple Words — Student is asked to read 10 provided words aloud.
• S3b: Writing Simple Words — Student is asked to write 10 words read aloud by the enumerator.
• S3c: Comprehending Simple Words — Student is provided with 5 words and a series of images, and is asked to match each word with the corresponding image.
• S4a: Reading Simple Sentences — Student is asked to read 4 provided sentences aloud.
• S4b: Writing Simple Sentences — Student is asked to write 3 sentences read aloud by the enumerator.
• S4c: Comprehending Simple Sentences — Student is provided 5 sentences and a series of images, and is asked to match each sentence with the corresponding image.

In past years, we have seen significant floor effects in conducting the EGRA test in Laos — a significant portion of students have scored zero points on a majority of test items. This makes looking at changes in literacy scores over the school year relatively uninformative for the subtasks measuring word-level and sentence-level reading ability. To supplement this with meaningful information, we will therefore look at another metric: zero reduction, or the change in the percentage of students that score zero points on any given section between the start and end of the year. Both changes in subtask scores and reduction in zero scores will be considered in evaluating the program.

PoP conducted baseline and endline testing in Control schools in order to observe learning gains in schools which did not receive EDF’s program. Both Treatment and Control schools conduct English lessons for the same amount of time each week (i.e., 1.5 hours for one day per week). All Treatment schools were in the Xam Thai district of Huaphan province, while all Control schools were in the Vieng Xay district of Huaphan province. The discrepancy in district location between Treatment and Control was due the terms of agreement between EDF and Huaphan government officials.

PoP tested a random sample of 7–8 students per grade across two grades (Primary 3 and 5) in 12 Treatment schools and 8 Control schools. This gives a total of 286 students across 20 schools that participated in both baseline and endline tests.

Given the rural nature of the communities that our schools serve, testing an appropriate sample of our students requires several weeks. Students are taken out of class one-by-one to avoid impacting instructional time for others. Baseline testing was held over October 11–27, 2017. Endline testing was held over April 24 — May 8, 2018.

Results

Key Takeaways

• Students in EDF-Lao schools with Momosign scored higher at the Endline test across half of EGRA sections than students in comparable schools without Momosign.
• Five subtasks (Letter Identification, Letter Writing, Letter Sounds, Reading Simple Sentences and Writing Simple Sentences) show that a higher Baseline score is a significant predictor of a higher Endline score.
• However, results show that a student scoring zero at Baseline is not predictive of their score at Endline. The exception is for two subtasks, Reading and Writing Simple Sentences.
• Many structural variables, including student gender, student ethnicity and socioeconomic proxies, such as access to water and electricity at home, do not significantly predict a student’s test performance at Endline.

Letter Identification vs. Letter Sounds: Student proficiency in letter identification at the Endline test, as well as relative growth in this proficiency from Baseline, is far greater than that of letter sounds. In other words, students can look at a provided letter and say what the letter is called, but often cannot produce the sound of the letter. Figures 1 & 2 below illustrate the score differences between the two relevant subtasks. This is consistent with recent conversations (c. June 2018) around how English is taught in Lao schools.

Figure 1: Distribution of S1a (Letter Identification)

 

Figure 2: Distribution of S2 (Letter Sounds) scores

 

Literacy vs. Comprehension: Endline data suggest that student ability in the Reading Simple Words and Writing Simple Words subtasks (titled S3a and S3b below) is far outpaced by that of Comprehending Simple Words subtasks (titled S3c below). In addition, significant growth in Reading and Writing Simple Words scores do not occur as a result of treatment, but does for Comprehending Simple Words. Put transparently, the Endline test data suggests that students are able to comprehend words (by matching them with a corresponding image), but students are less skilled at reading words aloud or at writing words read aloud to them. Figures 3, 4& 5 below illustrate these differences in scores with a series of histograms.

 

Figure 3: Distribution of S3a (Word Reading) scores

 

Figure 4: Distribution of S3b (Word Writing) scores

 

Figure 5: Distribution of S3c (Word Comprehension) scores

 

A similar effect is seen in sentence-related subtasks (see Figures 6, 7 & 8 below). This further supports the interpretation that the Lao syllabus and teaching method strengthens verbal comprehension in place of literacy.

 

Figure 6: Distribution of S4a (Sentence Reading) scores

 

Figure 7: Distribution of S4b (Sentence Writing) scores

 

Figure 8: Distribution of S4c (Sentence Comprehension) scores

 

Conclusion

Evidence suggests, with high confidence, that Momosign is effective in increasing students’ scores across five of ten test subtasks. The Momosign program seems to have a positive influence on simpler literacy tasks, such as Letter Identification and Letter Writing, and on comprehension-oriented tasks, such as Comprehending Simple Words and Comprehending Simple Sentences. However, Momosign did not significantly change performance on more complex literacy tasks, such as Letter Sounds (saying the sound of the letter, versus its name), Reading Simple Words, and Reading Simple Sentences.

The improved performance seen in comprehension tasks compared to reading tasks suggest two conclusions: 1) there is difficulty with the pronunciation aspect of reading (i.e., the comprehension task is non-verbal and requires only matching) and/or 2) students are memorizing word and theoretically matching a picture to a picture (i.e., seeing the written word “pencil” as a picture and matching it to the actual picture of a pencil). If these conclusions are in fact true, the teacher should place more emphasis on reading exercises and rely less on picture-matching activities.

We also found that when a student scores zero on the Baseline test it doesn’t serve as a predictor of their Endline score. This suggests that students who score zero are not left behind in the learning process and still have just as good of an opportunity to improve as students who have higher Baseline scores. Momosign also appears to be equitable between individual students, including across gender and socioeconomic levels.

As a result of this evaluation, there are remaining suggestions that could be pursued in order to better define the effectiveness of Momosign. First of all, it would be advantageous to conduct interviews with teachers in the Momosign program to gain a better understanding of how frequently program materials (e.g., books) are being used and how these materials are being used. Secondly, there is no information on the access to electricity in Control schools. If it were revealed that Control schools were significantly less likely to have access to electricity and, therefore, would not qualify as a potential beneficiary for Momosign, the group would be determined as incomparable. Lastly, schools in both the Treatment and Control groups were not randomly selected, which could create an argument for biased selection. However, since there were no differences in Baseline data, it cannot be determined how much the lack of randomization influenced the results.

PoP’s first opportunity to complete an external evaluation was an overall success. Our Learning & Evaluation (L&E) team in Laos, especially our Senior L&E Teacher Support Technician, Bounlam Khanty, did a phenomenal job in collecting high-quality data. As for the future of PoP’s use of Momosign in the classroom, our team is in the process of developing our own content, as we have found the use of projectors (a Momosign requirement) does not lead to the sustainable efforts we want to achieve. We are looking forward to the revamped approach we’ll be taking to Teacher Support in Laos during the 2018–2019 school year.