by Joshua Otlin, Principal – Milford High School @jcotlin
I am a first-year principal at Milford High School, a Level 3 school ranked at the 19th percentile. I am expected to help the school receive a Level 2 rating, so I have studied the system of ratings and rankings to inform my action plan. I want to share some of what I have learned because I am troubled at what I found: for the vast majority of schools, demography is destiny, as the above chart illustrates. In other words, level ratings and percentile ranks are almost exclusively a measure of the concentration of disadvantage in a school, not school quality.
My conclusions are drawn from the most recent accountability data on the 253 high schools with both a level rating and percentile rank. The tables and charts here include data from 208 “general” high schools and exclude 45 schools with selection-bias: charters, pilots, exam schools, and vocational schools (click here for the full data set, including all 253 high schools with a rating and a rank).[i] Including those schools adds more outliers, but does not alter the findings:
Schools with few Economically Disadvantaged (ED) students are immune from negative ratings
Every school with less than 25% ED students received a Level 1 or 2 rating, even if the school was outranked by Level 3 schools with more than 25% ED students. Conversely, there are schools with 25%+ ED students that match or outrank schools with less than 25% ED students but receive a lower rating. The following table shows the 16 schools with a rank between 21-30, the range where Level 2 and 3 schools overlap.
|Rank||School Name||Rating||ED %|
|30||Waltham Senior High||Level 3||35.6|
|29||Woburn High||Level 2||23.6|
|28||Westfield High||Level 3||26.6|
|28||Brockton High||Level 2||49.4|
|27||Rockland Senior High||Level 1||24|
|27||Methuen High||Level 3||31.9|
|27||Wareham Senior High||Level 2||41.9|
|26||Plymouth South High||Level 2||16.9|
|25||Gloucester High||Level 2||34.5|
|24||Attleboro High||Level 2||27.7|
|24||Worcester South High||Level 3||61.1|
|23||Leominster High School||Level 2||29.5|
|23||Everett High||Level 2||43.9|
|22||Plymouth North High||Level 2||22.5|
|21||E Brookfield (David Prouty)||Level 2||33.4|
|21||Salem High||Level 2||46.1|
A school’s concentration of poverty is strongly correlated with the school’s rank
This scatterplot of the 208 general high schools shows the clear relationship between poverty and rank (click here for an interactive version including all 253 high schools in the data set):
The association is very strong, but there are also numerous schools serving more than 25% ED students that outperform their demographic peers and received favorable ratings. Of course, this is also true for schools with less than 25% ED students, except all of those schools received favorable ratings. The schools serving 25%+ ED student and receiving favorable ratings surely have something to teach us about preparing disadvantaged students for grade 10 MCAS. They do not, however, change the fact that the odds of earning favorable ratings diminish dramatically as the concentration of disadvantage increases.
Racial achievement gaps are an illusion
An initial look at race suggests a strong association with ratings:
|Racial Composition||# of Schools||% of Schools at Levels 3-4|
A closer examination, however, reveals that the initial analysis is misleading. When we look at poverty, too, we see that race is not so tightly linked to rating.
We know that race is strongly associated with income in Massachusetts, so this finding is unsurprising. We do see, however, some important differences when we take a more nuanced look at race:
|Association with Rank (P < .0001)|
There is a relatively strong association between the concentration of Hispanic students in a school and that school’s rank, although not nearly as strong as the association between rank and ED students. Hispanics students, on average, are the most disadvantaged in Massachusetts: they are most likely to come from extremely poor households and a significant number are English-learners (see here for more on race and income in Massachusetts).
When we look at the intersections between race and poverty and accountability, we see that ratings and rankings simply reflect patterns of privilege and power in our society. As long as we maintain the status quo, demography will dictate results for the vast majority of schools. For a school like Milford, one of 39 general high schools with at least 25% Hispanic and 25% ED students, this spells trouble. 87% of these schools received a Level 3 or 4 rating and the median rank was 11 (when including schools with selection-bias and at least 25% Hispanic students, the median rank is still only 15). Knowing this, I will still do my best to help our school receive a Level 2 rating, but I do so with the understanding that my odds are quite poor.
While I remain opposed to the use of standardized tests for rating and ranking schools, I accept that there is little prospect of change and hope that system might at least be reformed to be marginally less bad. To do so, I suggest that we employ the “concentration of disadvantage” in the same manner that we use “degree of difficulty” in a number of sports. Schools with high concentrations of disadvantage work at a high degree of difficulty with regards to test-based accountability. The system, however, rests on the assumption that there is no such difference between schools. This is unrealistic and irresponsible. Developing a model that accounts for these difference would be no more complicated than analogous scoring systems in athletics.
Ratings and rankings do not tell us anything meaningful about school quality. They largely reflect the concentration of disadvantage in a school, and to a lesser extent for schools with 25%+ ED students, they tell us about the efficacy of the school’s MCAS preparation program. If I am wrong, however, and the system accurately measures school quality, then the data clearly demonstrates the failure of 20+ years spent trying to reduce achievement gaps through test-based accountability. Either way, it’s past time for something new.
About The Author
Joshua Otlin is a first-year principal at Milford High School, his alma mater. Prior to coming to Milford, he most recently worked as an Assistant Principal at Hudson High School, where he also served a data specialist for the district. Joshua has worked in Massachusetts’s public high schools for 18 years, teaching Social Studies before becoming an administrator.
[i] Comparing “general” high schools to schools with selection-bias is inappropriate and adds to the misinformation surrounding accountability measures, but I have conducted multiple rounds of analysis, both with and without schools with selection-bias, and my key findings remain unchanged. I encourage you to download the data set and see for yourself.
Write for the MSAA Blog
Are you interested in sharing your ideas, insights and questions with colleagues? You don’t need to be an experienced blogger. Consider reworking a piece you wrote for staff or families. For more information or to sign-up to write a post, contact firstname.lastname@example.org.