Educational Achievement Clinical Trial
The inability to consistently deliver at large scale promising education interventions is an
important contributing cause to inequality in the U.S. The research team applies insights
from price theory and field-based randomized controlled trials to examine the effect of
implementing a promising academic skills development program at large scale before
implementing at scale. The project is designed to provide evidence of direct scientific and
policy value for attempts to scale up a specific intervention, but also stimulate a much more
thorough investigation of social policy scale-up challenges by refining these methods and
demonstrating their feasibility and value.
The research team examines the challenge of program scale up for a promising intervention
studied in Chicago at medium scale in the past - SAGA tutoring. Past work has demonstrated
that SAGA's intensive, individualized, during-the-school-day math tutoring can generate very
large gains in academic outcomes in a short period, even among students who are many years
behind grade level. This study will explicitly explore the extent to which there is a
trade-off between effectiveness and scale for this intervention. By taking advantage of the
power of random sampling, this study will also allow for observation of the program's
effectiveness as if it were running at three-and-a-half times the proposed scale in a subset
of the study population.
The University of Chicago Education Lab and Crime Lab New York research teams are carrying
out a randomized controlled trial during the 2016-17 and 2017-18 academic years to build on
previous collaborations with the Chicago Public Schools (CPS), the New York City Department
of Education, and SAGA Innovations that have found that SAGA's intensive, individualized,
during-the-school-day tutoring can generate very large gains in academic outcomes in a short
period of time, even among students who are many years behind grade level. This research
suggests the promise of this approach for improving the academic skills and educational
attainment of disadvantaged youth, even once they have reached adolescence. However, to truly
affect outcomes at the local and national level, SAGA would have to be rolled out on a much
greater scale than researchers have been able to study in Chicago. Yet little is known about
how to take promising interventions to scale. This study seeks to build the science of
scale-up, by examining the extent to which this individualized tutoring program can be
implemented at an even greater scale and by explicitly exploring the trade-offs between
effectiveness and scale.
The SAGA Innovations program expands on the nationally recognized innovation of high- dosage,
in-school-day tutoring developed in Match Education's charter school in Boston. The tutoring
program meets as a scheduled course, Math Lab, once a day during the normal school day, and
is provided in addition to a student's regular math class. Students work two-on-one (two
students with one tutor) with the same full-time, professional tutor for the entirety of the
school year. The content of the tutoring sessions is aligned with what students are learning
in their regular math courses, but is also targeted to address individual gaps in math
knowledge. Also following the original model developed by Match Education, SAGA tutors use
frequent internal formative assessments of student progress to individualize instruction.
A previous randomized controlled trial conducted by the University of Chicago research team
found that one year of this intervention, delivered in AY2013-14 in the Chicago Public
Schools, generated between one and two extra years of academic growth in math, over and above
what the normal U.S. high school student learns in one year (Cook et al., 2015; Reardon,
2011). The estimated effects for math achievement are on the order of 0.19 to 0.30 standard
deviations, depending on the exact test and norming used. The intervention also improved
student grades in math by 0.58 points on a 1-4 grade point scale, compared to a control mean
of 1.77. These gains are particularly important because of the growing evidence on the
importance of math specifically for short- and medium-term success in school, and for
long-term life outcomes such as employment and earnings (Duncan et al., 2007).
This study aims to build upon the investigators' previous evaluations of the program, and
will provide insight into the ability of this program to serve youth at a much larger scale.
Specifically, this study aims to answer the following research questions:
1. What is the effect of implementing an evidence-based individualized tutoring program at
larger scale?
2. What is the relationship between the effect of the program and the scale at which the
program is implemented?
Implementation sites are divided into two sets: sites in Chicago at which students are
randomized to receive tutoring (hereby referred to as "scale-up" schools), and sites in
Chicago and New York City where principals have primary discretion to choose which students
receive tutoring (hereby referred to as "returning schools").
In order to study research question #1, investigators will take advantage of the power of
random sampling to study scale-up of this program without actually having to implement the
program at a much larger scale. The first research question seeks to measure the average
quality of the scaled up program in Chicago, in which researchers will utilize data from the
scale-up schools. Students in both scale up and returning schools are both randomly assigned
to tutors. However, students in the scale-up schools have two additional randomizations - (1)
tutor applicant randomization and (2) treatment assignment randomization. For the first
feature, the research team is having SAGA over-recruit tutors as though they were
implementing at larger than the intended scale in the scale up schools. Investigators then
randomly select one in three-and-a-half tutor applicants to continue through SAGA's standard
hiring process, and positions at the scale-up schools are only filled by these randomly
selected tutors. As students are randomly assigned to treatment in these scale-up schools,
investigators will be able to measure program effects at about three-and-a-half times the
scale at which the program is being implemented at these schools.
To measure treatment effects for research question #1, the research team will estimate both
intent to treat (ITT) and treatment on the treated (TOT) impacts. Researchers will estimate
the ITT effect as follows:
Y=B0 + B1T + B2X + E
where Y is the outcome of interest, T indicates students that are randomly assigned to be
offered the chance to participate in the tutoring program, X is a set of baseline controls
(which specifically includes randomization block, gender, age, learning disability,
free/reduced lunch status, race, baseline grade level, GPA, number of As/Bs/Cs/Ds/Fs in the
prior year, math and reading standardized test scores from the prior year, days absent from
school, disciplinary incidents including suspensions and arrests, and a binary flag for
students with missing GPA and attendance data), E is a random error term, and B0, B1, B2 are
parameters to be estimated. The random assignment of T assures that under standard
assumptions, Ordinary Least Squares (OLS) estimation yields an unbiased estimate of the ITT
as the estimate of B1, or the effect of being offered participation in the SAGA tutoring
program. As all students who were randomized into the program were paired with tutors that
were randomized via the process described above, our ITT (and subsequently, TOT) effect will
specifically measure the effect of the program when administered at three-and-a-half times
the scale as it is currently being administered.
The ITT measures the effect of being offered the chance to participate. As students assigned
to treatment are not required to participate in the program, the ITT may not measure the
effect of participation. The research team will measure the effect of participation using
random assignment of T as an instrument for participation. If all participants were randomly
selected (i.e. if there are no control students who are allowed to participate in the
tutoring program) this method calculates the effect of the treatment on the treated (TOT), or
the effect of participating for the group of students who choose to participate.
To gain insight into research question #2 above, which seeks to determine the relationship
between program scale and effects, tutors at all sites are ranked by SAGA leadership based on
relative expected quality. The research team will then randomize student pairs to tutors in
an effort to identify a tutor's effect on student outcomes. Using this methodology,
researchers can study whether tutor ranking predicts the size of the program effects. As the
research team assumes that the program would hire tutors in the order of their ranking
depending on the number of tutor slots they needed to fill, this analysis will shed light on
the relationship between scale and effectiveness.
To analyze the impact of being assigned to a tutor of a particular ranking (i.e. the ITT
estimate of research question #2), investigators will run regression models that regress
academic outcomes on tutor rank. Our main outcome of interest is math standardized test
scores. Our primary analysis will model outcomes as a linear function of tutor rank. As a
secondary exploratory analysis, we will estimate the shape of the relationship between
outcomes and tutor rank using the following leave-one-out cross-validation exercise:
1. Estimate the relationship non-parametrically by running a regression at the
student-level of the outcome (primary: math standardized test scores) on a full set of
separate fixed effects for every tutor rank. Call the coefficient on the fixed effect of
the rth ranked tutor, γ ̂_r.
2. For rank r=1,…,R: Estimate using the student-level data the relationship between the
outcome and tutor rank as a polynomial of order p=0, 1, 2, …, 10 holding the students
assigned to tutors ranked r out of the sample. Use the coefficients from the polynomial
terms to predict γ ̂_r, and call that f ̂_p (r). Save the squared error for each
polynomial: (f ̂_p (r)-γ ̂_r )^2.
3. Select the polynomial p that minimizes ∑_(r=1)^R(f ̂_p (r)-γ ̂_r )^2 .
4. Report the estimated relationship between the outcome and rank using the selected order
of polynomial.
In addition to the functions of tutor rank, each regression will include block fixed effects
that capture how student pairs were randomly assigned to tutors. The blocks include student
groups within a classroom with shared special restrictions (e.g. having no restrictions,
needing a Spanish-speaking tutor, or needing a tutor who is qualified for advanced
mathematics courses). Other covariates in the model will be the same as those included in our
ITT and TOT analysis for research question #1, noted above, to measure the effect of the
program when administered at three-and-a-half times the scale as it is currently being
administered.
As students switch tutors for a variety of reasons, researchers will also need to calculate
the TOT estimate to look at the impact of being assigned to and actually working with a tutor
of a particular ranking. To do so, investigators will use the rank of the randomly assigned
tutor as an instrument for the weighted average tutor rank, where the weight placed on each
tutor's rank is equal to the proportion of time (measured in days) the student spent with
that tutor. Investigators will use daily attendance data to create this weighted average.
This method will help investigators understand whether working with a higher-ranked tutor
means that a student actually receives better quality instruction, or whether an additional
relationship between tutor rank and actual quality exists.
We will use only observed outcomes for the analysis. If an outcome is missing for more than 5
percent of the sample, we will also report the treatment effect on whether or not the outcome
is observed. When baseline covariates are missing, we will impute missing values with zero
and include an indicator for missingness as an additional baseline control.
;
| Status | Clinical Trial | Phase | |
|---|---|---|---|
| Completed |
NCT04540003 -
School Based Health Care: A Model for Improving Educational Achievement for Children in Inner City Schools
|
N/A | |
| Completed |
NCT01678846 -
Good Schools Study
|
N/A | |
| Active, not recruiting |
NCT02673645 -
Remediating Academic Skill Deficits Among Disadvantaged Youth
|
N/A | |
| Active, not recruiting |
NCT01927523 -
Improving Life Chances of Disadvantaged Youth: Testing Best-Practice Academic vs. Non-Academic Supports
|
N/A |