Beware false idols of education excellence: take PISA test results with a grain of salt

Feb 4, 2021

Research shows that a large majority of students do not fully try on low-stakes tests such as PISA. There is little incentive for students to perform because there are no personal consequences. This could help explain the stark contradiction between Australia’s falling PISA results and its improving Year 12 results. 

The international PISA tests have become “false idols of educational excellence for the world to worship”. They have extraordinary status and influence. Education systems and even nations are judged by test scores and rankings that are assumed to be accurate measures of achievement. A growing literature shows this to be false.

The OECD report on PISA 2018 found that more than two-thirds of students in the OECD did not fully try on the tests. Large proportions of students did not fully try in Germany (80%); Denmark and Canada (79%); Switzerland (78%); and Australia and New Zealand (73%). In contrast, 46% of Korean students and 60% of Japanese students did not fully try. This variation calls into question the validity of league tables of countries based on PISA results.

A growing body of research literature shows that motivation has a profound effect on results in low-stakes tests such as PISA. Students don’t have much incentive to perform because there are no personal consequences depending on their results. Students and their parents are not given the results and they have no effect on academic careers.

A paper published in the Educational Research Review reviewed the results from 28 studies of test-taking effort and test performance conducted between 2005 and 2018. Most of the studies used low stakes tests. Nearly all found a statistically significant positive effect between test-taking effort and test results. Higher motivation and effort led to higher results; low motivation and effort led to lower results.

These results complement a 2005 meta-analysis of studies which compared results in low stakes tests with those of high stakes tests such as those required to complete Year 12. It found that students’ results in low stakes tests are lower than in high stakes tests.

These findings have major implications for international comparisons of test results such as PISA. As the OECD acknowledged in its PISA 2018 report, differences in student effort across countries will affect results and rankings.

A study published by the US National Bureau of Economic Research based on data from PISA 2015 found that differences in the proportion of students not fully trying had a large impact on the rankings for several countries. For example, it estimated that Portugal’s ranking in science in PISA 2015 would have improved by 15 places (31st to 16th) if students had fully tried. Sweden’s ranking would have improved 11 places (33rd to 22nd) and Australia’s by four places from 16th to 12th.

A significant gap in research is the extent to which student motivation changes over time in different countries and how it affects trends in the results of low-stakes tests.

In several OECD countries, including Australia, PISA scores have declined significantly since 2000 and reduced student motivation may be a factor. While there is no direct evidence, there is indirect evidence to suggest it is a factor.

The PISA results show increasing student dissatisfaction at school may show up in reduced effort and lower results. Student dissatisfaction among 15-year-olds in Australia has increased significantly since 2000. The proportion of students who feel unconnected with school increased fourfold from 8% to 32% between PISA 2003 and 2018. This was the third largest increase in the OECD behind France and the Slovak Republic.

The fact that one-third of Australian students are dissatisfied is likely to manifest in low motivation and effort in low stakes tests. The OECD says students who feel they do not belong at school have significantly lower levels of achievement in PISA than those who feel they belong.

The trend in Australia is repeated in OECD countries. PISA results for OECD countries have fallen since 2000 while the proportion of students who feel they don’t belong at school increased threefold from 7% to 22%. Of 30 countries for which data is available, all experienced an increase in student dissatisfaction at school and PISA maths results fell in 21 (see Chart).

Sources: PISA 2015, Vol 3, Table III 7.4; PISA 2018 Vol 2, Table II B 1.3.4; PISA 2018 Country Overviews.

New Zealand experienced similar increases in student dissatisfaction and similar large declines in PISA results to Australia. Canada, France and the Czech Republic experienced large increases in student dissatisfaction and significant falls in PISA maths scores.

In contrast, several countries including Austria, Denmark, Estonia and Japan had only small increases in student dissatisfaction and small statistically significant changes in their PISA scores.

This indicates that student dissatisfaction with school is not the only factor in declining PISA results. For example, some countries such as Finland, Korea and Netherlands had small increases in student dissatisfaction and large declines in PISA scores. Italy, Poland and Portugal achieved large increases in PISA scores despite modest increases in student dissatisfaction.

Clearly, other factors also influence test scores such as economic inequality, school funding, immigration, and shortages of teachers and learning materials. For example, Finland and Australia experience a similar decline in PISA results but the increase in student dissatisfaction at school for Finland was half that in Australia.

A large increase in the immigrant population and cuts in funding may have contributed to the decline in Finland. Australia’s results are likely to be affected by shortages of qualified teachers. For example, about 20% of maths teachers in Australia are not qualified to teach maths.

That student effort on PISA has declined helps explain the contradiction between Australia’s PISA and Year 12 results. Some 75-80 per cent of students participating in PISA are in Year 10. PISA results for these students have declined since 2000 while results for students two years later in Year 12 have improved significantly. The percentage of the estimated Year 12 population that completed Year 12 increased from 68% in 2001 to 79% in 2018 [Report on Government Services 2007, Table 3A.122 & 2020, Table 4A.70]. Year 12 assessments are high stakes even for less motivated students because these assessments have personal consequences for future education and work opportunities.

 

While more research is needed, the results of the 28 studies reviewed in the paper strongly indicate that international and national test results are significantly affected by differences in student motivation and effort. The results of these tests could be as much a measure of student effort as a measure of student learning. Much caution is necessary in interpreting results and drawing strong policy conclusions from these “false idols of educational excellence”.

Share and Enjoy !

Subscribe to John Menadue's Newsletter
Subscribe to John Menadue's Newsletter

 

Thank you for subscribing!