How much do peer-group rankings change when you change the peer-group definition?
A reproducibility log: the data this analysis touched, the queries it ran, the outside sources it leaned on, and the steps a third party would follow to re-run it.
Summary
Built three peer-group variants (k=20, k=40, k=80 nearest neighbors), ran the percentile calculation for every school under each variant, and computed the average shift in percentile rank between the variants.
Data sources
- Database tableschools
One row per New York City public school (DBN, name, district, borough, school_type, grade_band, latest_enrollment, proclivity_decile, proclivity_score, closed_at, admission_category).
- Database tablepeer_groups + school_peers
K-nearest-neighbor peer groups derived from grade band + admission category (hard match) plus demographics + topic similarity (weighted). Default k=40. Built in scripts/derive/peer-groups.ts.
Steps
Re-derive peer groups at k=20 and k=80; recompute percentile-within-peers under each; compute per-school absolute difference vs k=40.
Caveats
Sensitivity is a property of the rank metric, not a flaw in the schools themselves. The story argues against treating any single ranking as definitive.
Reproduce
scripts/derive/peer-groups.ts builds the default k=40 set; rerun it with k=20 or k=80 and recompute percentile_within_peers in the metrics summary table.
The recipe lives at data/stories/recipes.ts in the repo. Corrections welcome.