The Minister for Education, Alan Tudge, was fudging figures to denigrate Australia’s school performance at the The Age education summit last week. He claimed the UK as the new benchmark for education performance, but ignored serious flaws in the reporting of its results. He also fudged data on school funding and student results in Australia.
Tudge told the summit that the UK has dramatically improved its education results in the OECD’s Programme for International Student Assessment (PISA) over the last 10 years despite cutting school funding. However, the UK PISA 2018 results are significantly over-stated because of several flaws, most notably very low school participation and high student exclusions from the tests.
The flaws are exposed by Professor John Jerrim of the University College of London in a paper to be published in the academic journal Review of Education. His analysis comprehensively refutes Tudge’s claim. He says: “There is clear evidence of an upward bias in the PISA 2018 data for England and Wales, with lower-achievers systematically excluded from the sample”.
Jerrim estimates that the combination of student exclusions, school non-response, student non-response and technical details about eligibility criteria meant that about 40% of the target UK student population did not participate in PISA 2018. This was the fourth lowest participation rate of the 79 countries participating in PISA 2018. Only Panama, USA and Brazil had lower rates.
He shows that the PISA 2018 data for England (which accounts for 84% of the UK sample) clearly under-represents lower achieving students and over-represents higher achieving students. For example, 21% of the PISA sample were low achievers compared to 29% for the total population of the age group.
Another issue analysed by Jerrim is the school response rate. The OECD requires that 85% of sampled schools agree to take part in the study. However, both England and Northern Ireland failed to meet this standard with only 72% and 66% respectively participating. The overall rate for the UK was 73%. While there is provision to include replacement schools, the PISA technical criteria require a very high participation rate from such schools which was not met by the UK.
Even the UK Department of Education admits that that the UK “did not fully meet the PISA 2018 participation requirement” [p. 188] because of the high school non-response. However, the OECD waved this through and agreed that the UK data should be included as fully comparable to other countries. Jerrim says the OECD has a very weak adjudication process to decide whether to include a country’s data in the PISA results.
Jerrim also shows that the UK had a high rate of student exclusion from the tests. Students can be excluded from the tests in various ways. Schools may decide not to test some students included in the sample, others may be declared ineligible because they moved school between the time the sample was designed and the time the test was implemented, parents may not consent for students to participate and some students in the sample may be absent on test day.
The OECD technical standards state that within-school exclusions should total less than 2.5% of the target population and that the combination of school-level and within-school exclusions should not exceed 5% of the target population. The UK failed to meet these standards. The within-school exclusion rate was 3.3% and the total exclusion rate was 5.5%. Jerrim notes that a strict application of PISA’s data quality criteria would have led the UK to be removed from the study as it was from PISA 2003 for similar breaches.
These exclusion rates are much higher than for many other countries participating in PISA 2018. The average within-school rate was 1.4% and the total exclusion rate was 3%. The total exclusion rates in Japan and South Korea were 0.1%. Such differences are likely to bias cross-country comparisons of PISA performance.
The overall high non-participation in the UK has clear potential to bias its PISA results. It creates large uncertainty about the results. This is likely to affect the reliability of comparisons to other countries and how results have changed over time.
Jerrim estimates that the average PISA scores in England and Wales were inflated by the high non-participation rate in PISA 2018. The average PISA mathematics score for England was 504 points – significantly above the average across OECD countries. He estimates that had a truly representative sample of the population taken the tests England’s score would have been about 494. This is roughly the same as the OECD average and Australia’s score.
Jerrim concludes that the OECD adjudication process for deciding whether a county’s results should be accepted in a PISA cycle should be more transparent.
“… the OECD needs to reconsider its technical standards, the strictness of which these are applied, and its data adjudication processes…..the processes currently in place flatter to deceive and are nowhere near robust enough to support the OECD’s claims that PISA provides truly representative and cross-nationally comparable data.”
His analysis of the dodgy PISA results in the UK raises the broader issue of the validity of international comparisons when there are so many loopholes for countries to rig their results. He states:
“There remain many ways for countries to not test pupils who are technically part of the target population, with lower-achievers disproportionately likely to be removed from the sample.”
Apart from using the dodgy UK PISA results, Tudge also fudged school funding data in Australia in claiming it school funding per student, adjusted for inflation, increased by 60% since 2000. This is far from the truth. After adjusting for flaws in data from the Report on Government Services, we estimate the actual increase for all schools from 2001-02 to 2018-19 was only 19%, that is, an average increase of just over 1% per year. The increase for private schools was over double that for public schools. Government funding per student in private schools increased by 34% compared to only 15% for public schools. If school funding is failing to deliver better results as Tudge claims, this is mainly because money is being wasted on more privileged private schools instead of helping schools to overcome disadvantage in education.
Tudge also fudged Australia’s school results by highlighting the decline in PISA results for 15-year-old students and ignoring improving results in Year 12. The decline in PISA results is questionable because student motivation and effort is likely to be a factor in the decline. In contrast to Year 12 assessment, the PISA tests have no consequences for students as they don’t even get their results.
The OECD says that 73% of Australian students participating in PISA 2018 did not fully try in the tests. While there is no direct evidence of an increasing proportion of students not fully trying in the PISA tests over time, there is indirect evidence.
PISA data show that student dissatisfaction at school amongst 15-year-olds in Australia increased fourfold from 8% to 32% between PISA 2003 and 2018. This large increase in student dissatisfaction may have led to lower motivation and effort in PISA over time. The OECD says that the relationship between a feeling of belonging at school and performance in PISA is strong for students with the least sense of belonging [OECD 2016, p. 122]. Students who feel they do not belong at school have significantly lower levels of achievement in PISA than those who do feel they belong.
Tudge’s claim of declining school results is contradicted by other more significant data. The percentage of the estimated Year 12 population that completed Year 12 increased from 68% in 2001 to 79% in 2018, although there is an unexplained drop-off in 2019 [Report on Government Services 2007 & 2021]. The proportion of 20 to 24-year-olds who attained a Year 12 Certificate or equivalent increased from 79% in 2001 to 89% in 2019 [ABS, Education and Work, 2011 & 2020].
OECD data also shows that Australia had one of the larger increases in the OECD in the proportion of 25-34 year-olds who attained at least an upper secondary education. It increased by 19 percentage points from 71% in 2001 to 90% in 2019 [Education at a Glance 2002 & 2020].
These are indicators of an improving education system, not a deteriorating one. They are clearly inconvenient for Tudge because he ignores them and relies solely on questionable figures that mispresent Australia’s education performance.
Tudge’s fudges are designed to deny public schools the funding increases needed to ensure all students received an adequate education and to improve equity in education. Instead, the Morrison Government has provided billions of dollars in special deals for private schools and conspired with state governments though bilateral funding agreements to continue to under-fund public schools.
No more fudges, Mr Tudge. Your fundamental task as the Commonwealth Minister for Education is to better support public schools and disadvantaged students to deliver improvements in equity in education.