PISA 25 - an apology, sort of.

I decided to do a short update of my PISA 25 article after being attacked on social media by Sam Taylor of These Islands, an aggressive, dark-money-funded pro-Unionist think tank, a hydra head of the populist right.

Taylor disputed my suggestion that there was a bigger disparity in England between the students who took the test and the students who didn’t show up in terms of free school meals’ entitlement. (You can check out the argument here).

I was wrong on that - it seems that Scotland and England both have a similar disparity. The overall number of 15-year-olds registered for free school meals is also much lower in Scotland - although the system is more generous here. I think this is probably because most 15-year-olds in Scotland are allowed to go out of the school grounds for lunch, which they clearly enjoy.

Making a mistake in this complicated number juggling was upsetting. I should have checked more carefully. I was about to put it down to experience and walk away but then Taylor tweeted: “You clearly don’t understand statistics. Step away from your keyboard and stop embarrassing yourself.”

Reading the Room

I don’t like being bullied or humiliated so I went back to the numbers. I am sceptical of the way the PISA 25 results are being used. The Unionist press is one thing, we are used to that.

But some of what is happening is more international - many of these guys are not that interested in Scotland really. They are using England’s PISA 25 results to claim victory for a more traditional approach to education - attacking countries like Finland which has a very liberal model.

The PISA triumphalists claim this test is bad news for a ‘child-centred approach’. What is the alternative? It is that the child should not be treated as a partner in or a driver of their own learning. Obedience trumps well-being. Discipline trumps curiosity. Do what you’re told. Sit quietly. Don’t look out the window. Failure comes quick, for most. Better get used to it.

When I was young, only seven per cent of young people went to university. Many left school with few or no qualifications. There is a scene in a Bill Forsyth movie - is it Gregory’s Girl? - where the kids are in an exam hall looking at a paper where they can’t answer any of the questions. That was what traditional education looked like for a lot of children.

England has many more exclusions than Scotland. Is that the route to more disciplined classrooms and better PISA numbers? But what happens to the disappeared ones?

Continuing to dig around, I did find more reasons for scepticism. Riddle me this. Take a year group of 15-year-olds and test their attainment levels in a series of tests. With one group, you test them a few weeks into the autumn term, when they are just getting back into harness. Pupils coming back after the long summer holidays often show a drop-off in learning, that’s well known (maths is particularly affected).

With another group, you test them in the spring, when many are in revision mode for their first big exams, when they are at their most engaged.

So - which group would you expect to do better? Step forward England and Ireland. And which groups would you expect to show up as having months less schooling? Yes, that’s right, Scotland, Wales and Northern Ireland.

Defenders of the test will say that because PISA chooses a random group of 15-year-olds, about a third of the English sample was in the year below. But two-thirds were at the end of the fourth-year syllabus and about to take their GCSEs.

The Northern Ireland puzzle

Could this explain why the Northern Irish cohort that showed up on PISA 25 as being months behind their English counterparts went on to do a lot better in the GCSES? (The alternative explanation that Northern Irish GCSEs must be marked more leniently doesn’t hold water, because they are the same qualification, overseen by the same quality assurance framework).

It is not just me highlighting the difference in time of sampling - it was actually Professor of Mathematics Education at Southampton University, Christian Bokhove. He argues that PISA should be a starting point for comparison rather than a ringing endorsement of any approach.

Reading the Runes

I agree. I recently read a book about the Vikings called The River Kings by Cat Jarman. In it she tells the story of how carbon dating in the 90s revealed that the skeletons in a burial mound in Repton were hundreds of years older than previously thought. For the next twenty years, archaeologists tried to reframe their understanding to accommodate this new evidence - did Vikings bring the bones of their ancestors when they travelled? But then new research showed that eating a largely marine diet affects the carbon date (basically because the carbon in the sea is older).

So carbon dating has given us a lot of interesting insights into the past. But you have to be careful with data - especially when it is about people, who are complicated and do things that mess with the data, like eat different diets.

For example, Scotland’s child poverty rates are now significantly lower than in England, which, especially in the north, has some of the highest child poverty rates in the developed world. Our understanding of the effects of poverty would suggest that would filter through into attainment - not here apaprently. But it would not, be safe to conclude, as some commentators have done, that PISA shows that English educational reform is overcoming the effects of child poverty.

The PISA tests are not useless. As I argued the other day, they reveal things that are worth investigating further and no doubt Scotland, like other countries, can identify areas for improvement. But they are being weaponised by a global movement that has other things on its agenda and we should be sceptical about that too.

Figures for child poveryt in Scotland and England are here and the UNICEF report is here