With 287 peer-grouped schools and k=40, the rankings barely move at k=20

Summary

Ran the peer-group deriver at k=20, k=30, k=40 (all in one script execution). For each school, compared top-10 peer set membership between k=20 and k=40. Counted schools with ≥3 different members as a 'substantive flip.'

Data sources

• Database tablephilly_schools

  One row per Philly school. PK is ulcs_code (SDP canonical, = id_eos in PSSA, = SRC_School_ID × 10). Carries aun_code + pa_code for PDE-side joins.

• Database tablephilly_comparison_groups

  K-nearest-neighbor peer groups (k=20/30/40 derived; k=40 default). Hard filters: school level + admission bucket + sector. Demographic Euclidean across 7 features.

• Loader scriptscripts/loaders/philly/derive_peer_groups.ts

Steps

1. All three k values derived in one script call. k=40 is linked into philly_schools.comparison_group_id as the default per the integration plan.

2. Flip-diff: for each school, take the top-10 peer set under k=20 and k=40, count common members. Schools with <8 common are 'substantive flips.'

   -- Per-school top-10 peer comparison reads from philly_comparison_groups.methodology_notes JSONB
   WITH g20 AS (SELECT cg.id, cg.methodology_notes->'peers' AS peers FROM philly_comparison_groups cg WHERE cg.k = 20),
        g40 AS (SELECT cg.id, cg.methodology_notes->'peers' AS peers FROM philly_comparison_groups cg WHERE cg.k = 40)
   -- The TS deriver script does this comparison directly.

Reproduce

Run `npx tsx scripts/loaders/philly/derive_peer_groups.ts`. Output prints the flip-diff statistic at the end.