At the beginning of the 2018-2019 school year, Pencils of Promise (PoP) developed a version of the ASER to serve as an improved student literacy assessment. As outlined in an earlier Transparency Talk, the Learning & Evaluation (L&E) team at PoP identified the ASER to be the best option for assessing early literacy skills, especially for students who are learning English as a foreign language (PoP’s focused work in Laos). Recent test results confirmed that the ASER is in fact a better instrument for measuring the effect of PoP Teacher Support (TS) on student learning, which solidifies confidence in moving to the assessment tool.
Photo credit: Justin Xaisanasy
In preparation for the 2019-2020 school year and expansion of Teacher Support (TS) programming into the 5th grade, PoP’s L&E needed to develop an updated ASER to align with grade-appropriate curriculum. This served as an opportunity to refine the item selection methodology used for 3rd and 4th grade versions, especially in terms of selection of words and images, in order to advance the tool’s validity. At the same time of this planning process, PoP established another partnership with the School of International and Public Affairs (SIPA) at Columbia University and designed a master’s student capstone project that would produce a well-researched and refined 5th grade PoP ASER.
SIPA’s team of graduate students conducted an extensive amount of desk research and interviews to establish an understanding of what theoretical logic was missing in the original method for PoP ASER development. This included but was not limited to: interviews with experts in reading development from Columbia University and the University of Southern California, extensive dissection of the Lao curriculum for English in the 5th grade and a comprehensive literature review on critical characteristics necessary for successful early reading education. Results from their preliminary research led to the following central conclusions on adaptations necessary for an improved PoP ASER:
- Assess receptive vocabulary through object and image recognition in order to test foundational components of literacy
- Continue assessing productive vocabulary with letters, words and phrases
- Test multiple versions of the assessment to gauge differences in student performance/understanding
Three versions of the assessment were created that were scaled based on difficulty (Version 1 being the easiest and Version 3 the most difficult) and all were piloted in Laos during one week of testing in March of 2019. Based on the team’s research, assessment versions included a combination of the following sections: object recognition, image recognition, reading of words, reading of letters and reading of phrases. During the pilot testing period, a total of ten schools (five schools receiving PoP TS and five that were not included in the program) were visited and 126 students assessed.
Results from the pilot validated the difficulty of each version (Table 1) (i.e., students had the best scores on Version 1 and struggled the most with Version 3) and confirmed that object recognition was the easiest section while reading phrases was the most difficult (Table 2).
Arguably, the most important finding from the SIPA team’s work was the identification of objects, images and words that were either too difficult or confusing for the students. For example, the team recommended that pen not be used as an assessed object because students often confused it with a pencil. Additionally, the team found it important to include the same content in succeeding sections in order to help identify if and when a breakdown in recognition occurred. For example, a ball should be included as an object, an image and a word. During the assessment, if a student were able to identify the ball as an object and an image, but not as a word, a conclusion could be drawn that the student had not yet fully developed their productive vocabulary (when taking other assessed items into consideration).
Photo credit: Timmy Shivers
These high level results only skim the surface of what the SIPA team produced. PoP will continue working with the SIPA team (now graduates) with plans of submitting this body of work for peer review in a well established education journal later this year. This partnership helped the L&E team establish a sound methodological framework, which has already been applied to improve earlier PoP ASER tests (e.g., 3rd and 4th grade) for future use.
As the 2019-2020 school year begins in early September, PoP teams are well-positioned to continue delivering high-quality programming coupled with valid and reliable evaluations. PoP continues to strive for ongoing improvement and refuses to settle on sufficient service delivery. There is only the goal of ensuring PoP served communities and schools receive the highest quality education services with an everlasting effort to improve through refinement and adjustment.
Citation: Aggarwal, A., Ruiz, M.F.A., Brizio, L., & Lin, X. (2019). Development of an English reading assessment tool for 5th grade students in Laos (Unpublished masters capstone thesis). Columbia University, New York City, United States of America.