Key Takeaways
-
The Practice Makes Perfect (PMP) summer school model offered high quality teaching, small class sizes, and a team of teachers and mentors for each student.
-
Despite a promising program design, due to low program enrollment, LEO could not confirm statistically significant effects of the program on student outcomes.
-
It is possible that students who would truly benefit from the program were part of the group of students who did not participate in PMP.
Mechanics:
Practice Makes Perfect (PMP) operates a suite of summer learning programs for students from low-income communities in the New York City area. PMP is an innovative alternative to summer school that aims to reduce summer learning loss by providing both academic guidance and peer mentorship as part of their summer education program. The inclusion of both peer and instructor mentorship seeks to uniquely address the disparities in experiences, exposure, and access that exist between middle- and low-income students, and sets PMP apart from traditional summer school programs.
LEO collaborated with PMP to implement a randomized controlled trial (RCT) to evaluate the program’s impact on academic outcomes. The study team implemented the initial pilot phase of the RCT in the summer of 2018. This pilot was intended to lay the foundation for a larger-scale impact evaluation in the summer of 2019.
The study targeted schools that serve predominantly low-income students and that had capacity to provide PMP to up to 40 students during the summer of 2018. For the research project, PMP identified four schools that satisfied these criteria and were interested in participating in an impact evaluation. The initial plan for study enrollment was as follows. School administrators would provide a roster of at least 120 eligible students per PMP class offered. After sorting each group of 120 plus students into a randomly assigned rank order, the first 60 students from the list would be offered a spot in PMP, with the remaining students assigned to the control group. 60 students would be offered a spot in the program with the goal of reaching a class size of 20. The expectation that in most of the participating schools only 1-in-3 students would take up the offer was based on advice from PMP staff and on the fact that take-up is often low in field experiments. For three of the eight classes, PMP anticipated much higher enrollment rates based on their experience with those specific grades and schools. Accordingly, we estimated that 1-in-2 students would take up the program spot in those classrooms and planned to randomize 40 students into treatment for each of them rather than 60.
In practice, study program enrollment numbers fell short of those specified in the initial plan for two reasons. First, for half of the classes, fewer than 120 eligible students were provided by the school administrator. Second, the actual enrollment rate during the pilot study of 21% was much lower than the anticipated rate of 33-50%.
LEO assessed the impact of PMP by comparing academic outcomes, such as test scores and on-time progression to the next grade, for students in the treatment group to those for students in the control group using data from the New York City Department of Education. Because program take-up fell well short of what was anticipated, the research team was not able to estimate precise effects of the program.
What we learned:
To accurately measure the impact of a program, it is necessary to predict eligible students’ program take-up as accurately as possible and to reach the enrollment goals laid out by the research team during the project planning stage.