Reliability of National Standards assessments called into question

2015

A recent report has cast doubts over the reliability of teachers’ National Standards assessments and rises in student achievement.

exams broken pencil

It found that just 40 per cent of overall teacher judgments (OTJs) matched the ratings generated by the Progress and Consistency Tool (PaCT), a tool developed to improve the consistency of teacher judgments on National Standards assessments.

The PaCT was launched in schools in last year with a mathematics pilot. The latest report of the National Standards: School Sample Monitoring & Evaluation Project 2010-2013 compared the judgments from the pilot with a selection of teacher judgments finding significant variance.

However, the primary school sector has expressed concerns about PaCT.

New Zealand Principals’ Federation (NZPF) president Denise Torrey said the tool had “no place in a twenty-first century classroom”.

“Any tool based on national standards, whose flaws have never been resolved, is not going to offer helpful information to advance children’s learning,” said Torrey.  

Primary teachers’ union NZEI Te Riu Roa president Louise Green agreed that the PaCT is not a reliable measure.

“This is because a computer tool which pigeonholes a child's achievement cannot replace the professional judgment of a teacher working alongside a student on a daily basis,” said Green.

However, the report stated that teacher judgment data cannot be taken as evidence that student achievement is improving.

Increases in student achievement have been reported for students rated ‘at’ or ‘above’ the reading, writing and mathematics standards since National Standards were introduced. For example, 72 per cent of students were rated ‘at’ or ‘above’ the reading standards in 2010, and this increased two per cent a year, to 78 per cent in 2013. Substantial increases in the proportions of students rated ‘at’ or ‘above’ were also observed for some demographic sub-groups, such as Pasifika students in reading and mathematics, Year 7 and 8 students in writing and mathematics, and students at low decile schools in reading.

The report suggested that such increases should be treated with caution as these represent changes in teacher judgments over time rather than an actual shift in achievement.

Making sense of assessment

National Standards: were introduced in schools with pupils in Years 1 to 8 in 2010. The standards set expectations for the reading, writing, and mathematics knowledge and skills students need to achieve at each level of the New Zealand Curriculum.

Overall Teacher Judgments (OTJs): involves drawing on and applying the evidence gathered up to a particular point in time in order to make an overall judgment about a student’s progress and achievement. OTJs involve combining information from a variety of sources, using a range of approaches.

Progress and Consistency Tool (PaCT): an online tool developed to support teacher judgments launched in schools last year with a mathematics pilot. It captures a series of teachers’ judgments, turns that into a PaCT score range, and recommends an overall judgment which a teacher can confirm or review. The PaCT also provides frameworks that break down and illustrate aspects of mathematics, reading and writing.

National Standards School Sample Monitoring and Evaluation (NSSSME) Project:  In 2010 the Ministry of Education commissioned a three year monitoring and evaluation project to get an overview of the implementation of National Standards. The NSSSME is contracted to independent evaluation company Maths Technology Ltd and led by Dr Gill Thomas.


Post your comment

Comments

  • I wonder how many teachers feel that they are given the time to assess their pupils accurately, against the minute details of the curriculum? School leaders should perhaps take this in to account...?

    Posted by Andy R, 06/07/2015 9:40am (2 years ago)

  • Until all schools are assessing and reporting in exactly the same way, with built-in quality assurance any reported results will have to be treated with a degree of scepticism.

    Posted by Andy R, 06/07/2015 9:37am (2 years ago)

RSS feed for comments on this page | RSS feed for all comments