PoP has pretty big goals. We like to talk about PoP as a data-driven, emergent learning organization. Part of being data-driven is setting goals, then being able to collect information to measure our progress against those goals. Part of being responsive and adaptable, however, also means that those goals can (and sometimes should) shift as our programs evolve and as we learn more and more about our impact.
That’s exactly what’s been happening at PoP in the past months. Through the efforts of a special working group — with participants from every department! — we’ve thought long and hard about our current slate of Key Performance Indicators (KPIs), and we now have some refinements to announce.
A working definition
Our working group covered a lot of ground. Here’s just a small fraction of the discussion points we covered:
- What is a KPI supposed to achieve operationally?
- What is PoP’s goal?
- Should PoP have one goal or many to account for our various programs?
- What are key indicators of our progress toward that goal?
- Are our goals realistic?
- Are our goals aligned with organizational strategy for the future?
Staff responses to the question, “What determines our success?”
And some conclusions:
- PoP’s organizational goal is to promote student literacy.
- Our programs may have more specific goals (e.g. teacher behavior change for Teacher Support, student health behavior change for WASH), but all feed into the larger organizational goal of promoting literacy
- Our KPIs should focus on the larger organizational goal
- Other program indicators will continue to be collected and published, but will not be considered full PoP KPIs.
With these questions in mind, it was time to critically review our current KPIs and see which would be left standing.
Case Study: Build Index
We first introduced the Build Index in an effort to measure the effect of a PoP build in a community. Before we broke ground on a build, we would conduct surveys of teachers and students, asking them a battery of questions along four main themes: Motivation, Focus, Pride and Physical Space. We would then return after the build was complete and ask them the same questions. Each response was given a numerical rating, which we then aggregated into theme-specific and overall scores.
Build Index scores for Ghana 2016–2017
What we found was that Build Index scores increased by 26% on average in Ghana in 2016–2017. Great!
Now what?
With our newfound perspective on KPIs, a few weaknesses came to light:
- Interpretability. What did a 26% increase in the Build Index mean? What % increase should we be aiming for? Given that we had constructed this composite measure ourselves, we lacked a frame of reference to compare this against.
- Alignment with Goal. We can say that we’ve measurably increased the quality of the classroom setting and empowered teachers and students to do more in school. But what did this mean for literacy outcomes?
- Operational usefulness. What would we change based on what the Build Index was telling us? (Very little.) What would a change in percentage increases in any of the four main themes mean for our school builds? (Hard to say.) Would we stop building if the Build Index started reporting decreases in scores? (Very unlikely.)
We’ve decided to demote the Build Index to a mere Builds indicator. We’ll still continue with the survey — the component questions around specific themes may offer more interesting data for future analyses. But for now, it’s no longer a KPI.
What’s left standing?
To reiterate, PoP’s central focus is promoting student literacy. How do we get there? By empowering teachers, providing safe learning environments, encouraging healthy behaviors that reduce out-of-school days. All of this contributes to that central goal. And our refined set of KPIs are meant to reflect just that.
Slide from a staff presentation announcing the new literacy-focused KPIs
We’ve chosen to focus on reporting our progress against measurable literacy outcomes. To that end, we focus on different skills at each stage of literacy acquisition.
- Passage Reading Fluency. Specifically, this is the percentage of students in PoP schools meeting proficiency benchmarks for passage reading — 45 correct words per minute (CWPM) in English and 60 CWPM in Spanish. This will be the primary indicator of literacy for Transitional Readers, in Primary 3 and 4.
- Passage Comprehension. As above, this is the percentage of students in PoP schools meeting proficiency benchmarks for passage comprehension — being able to read a passage, then answer 80% of questions about that passage correctly. This will be the primary indicator of literacy for Independent Readers, in Primary 5 and 6.
- An early grade indicator for Beginner Readers (Primary 1 and 2). Unlike the higher grades, we believe that the strongest indicator of preparation to acquire important literacy skills is a child’s foundation in social and emotional skills — how well they engage with teachers and peers, if they have the confidence to participate in the classroom environment, and so on. We will draw on indicators from our successful Social & Emotional Learning (SEL) program here.
These are the measures that we want to be held accountable to — the indicators that will tell us if we’re truly achieving our goal of literacy for all.
Moving forward
While the new set of KPIs brings much-needed clarity over our organizational goals, there’s still work to be done and open questions to be considered:
- Expectations. Is literacy really achievable at the primary level in schools which we work? Should PoP look at longer-term programs that include secondary school?
- Standardization across countries. As our programs start to diverge, is it reasonable to apply the same literacy indicators in different country contexts?
- Early Grade indicator, TBD. We’ll dig deeper into our SEL program and select an indicator that best represents how well-prepared our early grade students (in Primary 1 and 2) are for acquiring literacy skills in later grades.
Each of these questions opens a door to a far deeper investigation. To do so, we’re looking forward to collaborating with staff, experts from our Graphite Impact Council and other literacy-focused organizations.
We’ll continue to document our journey as we try to answer these tough questions and more. Watch this space.