School evaluation is an essential component of the education system. Identifying high-performing and underperforming schools is vital for building an evidence base for what works to improve education outcomes for students. However, the task of evaluating schools objectively and without bias is extremely complicated.

The complex interactions between family socioeconomic background and academic performance result in measures of student academic achievement that are a mix of family, teacher and school effects. National Certificate of Educational Achievement (NCEA) league tables - which only show absolute measures of student achievement - tells us as much about students and their parents as the quality of the school.

When 35% of year 12 students in a school receive an Excellence endorsement at NCEA level 2, is that the result of a highly effective school that gets its students to meet and exceed their potential, the result of higher socioeconomic students enrolled in that school, or a combination of both?

A solution many countries have adopted is value-added models of assessment and evaluation. Value-added models differ from other methods of assessment in that they use a mix of prior student achievement data and family socioeconomic background data to separate the contribution of family background from the contribution of the school. This allows the Ministry of Education to identify what proportion of student achievement can be attributed to the school and to the family.

Value-added models also allow the Ministry to identify how effective one individual school is compared with every other school in the country, and which characteristics or institutional practices are associated with effective schools. The exact value-added model implemented varies depending on the purpose, stakes, political climate, and data availability in each country.

New Zealand does not have a nationwide value-added model, but it desperately needs one. Fortunately, New Zealand has an opportunity to build and implement one through Statistics New Zealand’s Integrated Data Infrastructure (IDI) – New Zealand’s largest research database. Better yet, because of the world-leading data available in the IDI, The New Zealand Initiative’s value-added model improves upon existing models significantly in both the breadth and depth of data it uses to identify and separate the contribution of family background.

Over the past year, the Initiative has done just that – build New Zealand’s first contextualised value-added model – alternatively called our school performance tool.

Using linked administrative data from the Ministry of Education, New Zealand Police, Ministry of Social Development, Department of Corrections, Ministry for Children, Immigration New Zealand, Inland Revenue, and the 2013 Census, the Initiative’s school performance tool is able to identify how much each secondary school contributes to its students after separating the contribution of each student’s family socioeconomic background.

This has allowed us to fairly and objectively compare low- and high-decile schools for the first time in New Zealand.

The results from our school performance tool indicate that the differences in school performance typically seen in NCEA league tables largely reflect the differences in the communities those schools serve, not large differences in school quality or effectiveness. 

Additionally, once adjusted for differences in family background, the large performance differences between deciles disappear; however, high-performing and underperforming schools still exist across all deciles. IN particular, when evaluated on University Entrance, 42 decile 1 and 2 schools outperform 75% of every other secondary school in the country; in contrast, 9 decile 9 and 10 schools are at the bottom 25% of all secondary schools in the country.

Furthermore, accounting for differences in family background, approximately 80% of schools perform almost identically when evaluated across a wide range of NCEA metrics.

The purpose of the Initiative’s school performance tool is to show what can and should be done with the world-leading data available in the IDI. The insights gained from our tool should be provided to all schools, principals and boards of trustees by the Ministry of Education.

If used by the Ministry, our tool could provide annual reports to every secondary school in the country – providing them fair and objective information on how they are performing.

Our tool could also be employed by the Education Review Office (ERO) or by the potential regional education hubs as recommended by the Tomorrow’s Schools Taskforce report. New Zealand needs fair and objective school evaluation. Without it, many top-performing low-decile schools will remain unrecognised, while many underperforming high-decile schools continue to fly under the radar.

The Initiative’s school performance tool provides the missing piece of the New Zealand school evaluation puzzle – fair, objective and data-driven information.

A summary of the policy recommendations from In Fairness to our Schools, is shown below:

1. Issue annual reports. Annual reports containing insights gained from our school performance tool should be provided by the Ministry of Education to every principal and school board of every secondary school in New Zealand. This will require demand from parents and for additional resources to be directed by the Minister of Education.

2. Allow the identification of individual schools in the IDI. This can be achieved by revising rule 5.14.2 of Statistics New Zealand’s Microdata Output Guide; reinterpretation of the Memorandum of Understanding and the Statistics Act 1975; or amendments to the Act.

3. Further development of our school performance tool in the IDI by the Ministry of Education. Any further research should be made open-source, identical to what we have done for all the coding for this project.

4. Implementation of our tool as part of an evaluation framework in any new government education policy through the Ministry of Education.

5. ERO to investigate any differences in institutional practice between low-, middle- and high-performing schools as identified by our tool. This would be in addition to further research comparing the Education Review Office’s conclusions with the conclusions gained from our tool.

6. Integration of Te Rito, Edsby and Novopay data into the IDI to allow more comprehensive evaluation of school performance in the future.

Read the two-page report summary of In Fairness to our Schools: Better measures for better outcomes, here.

Technical Paper: Separating school and family

Joel has written a technical paper to go alongside his research report, In Fairness to our Schools. It explains, in further detail, how he evaluated the effects of school and family background on student performance in NCEA. You can download the technical paper here.