My concern with the recent PISA scores debate is not that “PISA doesn't matter”; it is that some observers repeatedly make PISA say things PISA cannot tell us.
I am not in any way research savvy, but think some essential misunderstandings/errors include:
1. Converting a PISA score into a fictitious educational timescale- "19 months behind” sounds as though PISA has established that today's S3/S4 pupil is working at the level of an S1 pupil. It hasn't. If my understanding is correct, PISA points are a psychometric scale. The points-to-months conversion is derived from an average association between PISA performance and additional schooling; it is not a measure of an individual child's curricular progress. Some commentators have turned what is an illustrative statistical conversion into an apparently concrete educational fact.
2. Some observers believe PISA as an unchanged measuring instrument across decades. I don't think it is? PISA's frameworks, domains, item design and mode of assessment have evolved substantially, including the move from paper to computer. For this reason, the OECD uses sophisticated statistical linking precisely to make trend comparisons possible, but also acknowledges additional linking error between cycles. I fully agree that long-run trends are more than useful evidence; but not equivalent to administering the same test with the same ruler in 2009 and 2025.
3. Some observers appear to mistake association for causation- arguably a central failure? Scotland introduced CfE; Scottish PISA performance subsequently declined; therefore CfE caused the decline. That simply does not follow. If PISA is repeated cross-sectional observational data it cannot isolate the effect of curriculum from a range of other factors- e.g. assessment reform, qualifications, accountability, teacher supply, deprivation, attendance, demographics, ASN/SEND, Covid and/or numerous other changes.
4. Some observers appear to construct a false Scotland–England natural experiment.
England isn't Scotland with one variable (curriculum) changed. Both education systems differ in governance, academisation, inspection, accountability, examinations, league tables, selection, exclusions, SEND/ASN practice, school structures and so much more. Just by calling curriculum the decisive difference doesn't necessarily demonstrate that it is?
One particularly important given is that some observers/ commentators treat England's PISA result as though it were an unproblematic comparator. It might appear this is not the case? England's own 2025 analysis found participation was significantly related to school attainment: pupils in the lowest-attaining schools were less likely to participate and those in the highest-attaining schools more likely to participate. Pupils in selective schools were also more likely to participate. England initially achieved only 78% participation among originally sampled schools and used replacement schools to reach 90%.
To be precise, I am not arguing, as some have, that this proves deliberate English “gaming” of the processes. But surely this means we cannot simply treat the achieved English sample as an unbiased experimental control for Scotland.
And it would appear this isn't merely a theoretical concern. My understanding is that the OECD's investigation of the 2022 English/UK sample found residual upward performance bias associated with non-response- approximately 0.07 SD in reading and 0.09 SD in mathematics.
5. Too many observers caricature CfE into a simplistic trope of “skills instead of knowledge”. Anyone at the chalkface of our system knows this is simply not an accurate portrayal of Scottish secondary education. Secondary S1/2 CfE contains significant disciplinary content, and, as you know, our Senior Phase involves subject-specific National Qualifications with prescribed content and national assessment. For example, a 15-year-old Scottish pupil taking National 5 Chemistry, Maths or History is not sitting in a classroom acquiring generic “skills” instead of knowledge. If observers/ critics claim that specific knowledge deficit caused the PISA decline, surely they need to identify the missing knowledge, demonstrate that CfE 'implementation' removed or reduced it, and evidence that this deficit explains the PISA performance difference.
6. Some observers appear to misunderstand the very instrument they are using to make their argument. PISA (deliberately) is not a straightforward test of mastery of a national curriculum, rather it assesses/ measures assesses pupils' ability to apply knowledge and skills, reason and solve problems in contexts beyond simple reproduction of taught curricular material. A low PISA score does not diagnose in itself “insufficient knowledge”. Some commentators appear to have inserted this 'diagnosis' themselves.
7. We need to be careful as to how we explain PISA's rankings. "England sixth; Scotland 21st” sounds dramatic but may deliver spurious precision? I think PISA scores are estimates with confidence intervals; many countries separated in any ordinal 'league table' are not statistically distinguishable. The meaningful questions concern scores, distributions, effect sizes, trends and uncertainty- not whether a country happens to 'occupy' box 16 or 21 in a table someone has constructed.
8. Some observers/ commentators are a bit too keen to turn what is in essence an attainment-distribution observation into political rhetoric. If Scotland's socio-economic attainment gap has narrowed partly because higher-SES pupils' performance has fallen then this is clearly genuinely important and deserves investigation; but calling it, as some have, an “attack on the middle class” is not on any way an 'empirical' conclusion. PISA on its own surely cannot establish either governmental intention or any causal mechanisms responsible.
My own evidence is that Scottish secondary schools deliver a subject (knowledge)-based, schema focused, teacher-led/ student-centred approach to learning (with obvious inherent inconsistencies- as revealed in ongoing HMIE reports).The 'knowledge debate' is one that exists in our education quangoland and not at the chalkface. Indeed, even after all this time it is difficult to clarify exactly what CFE actually 'means'. Stating that CfE is the cause of our concerns is lazy at best if we do not define what CfE actually represents.
There are important issues raised by the latest PISA data, but my fear is we return to a short term cycle of blame (often with an anti-Scottish, anti- comprehensive bias) and lack of serious analysis over what needs to be done to identify improvements needed. Yes we MAY need to look at our delivery of reading/ literacy skills in primary; yes we MAY need to examine in more detail efficacy of teacher pedagogy; yes we MAY need to review the present curriculum framework design proposals away from the project- based learning trend that appears to be emerging . . .
'A plague on all your houses'. My worry is that, based one one narrow set of 'problematic' data, confidence of our profession takes yet another media hammering from the tit-for-tat 'dialogue' across groups who have their own self-interested 'skin in the education system game' or have their own ideological agendas, but are too far removed from the realities of classroom delivery to have any impact on pupil attainment.
Sorry for the detail, but needed to 'vent' :-) . . .
And this is exactly why I subscribe to your Letter From Scotland. Thank you
Thank you very much 🍻
Lots of nuance in this article and in the comments posted. A useful contextual corrective to the sloppy coverage in the media.
Thank you!
My concern with the recent PISA scores debate is not that “PISA doesn't matter”; it is that some observers repeatedly make PISA say things PISA cannot tell us.
I am not in any way research savvy, but think some essential misunderstandings/errors include:
1. Converting a PISA score into a fictitious educational timescale- "19 months behind” sounds as though PISA has established that today's S3/S4 pupil is working at the level of an S1 pupil. It hasn't. If my understanding is correct, PISA points are a psychometric scale. The points-to-months conversion is derived from an average association between PISA performance and additional schooling; it is not a measure of an individual child's curricular progress. Some commentators have turned what is an illustrative statistical conversion into an apparently concrete educational fact.
2. Some observers believe PISA as an unchanged measuring instrument across decades. I don't think it is? PISA's frameworks, domains, item design and mode of assessment have evolved substantially, including the move from paper to computer. For this reason, the OECD uses sophisticated statistical linking precisely to make trend comparisons possible, but also acknowledges additional linking error between cycles. I fully agree that long-run trends are more than useful evidence; but not equivalent to administering the same test with the same ruler in 2009 and 2025.
3. Some observers appear to mistake association for causation- arguably a central failure? Scotland introduced CfE; Scottish PISA performance subsequently declined; therefore CfE caused the decline. That simply does not follow. If PISA is repeated cross-sectional observational data it cannot isolate the effect of curriculum from a range of other factors- e.g. assessment reform, qualifications, accountability, teacher supply, deprivation, attendance, demographics, ASN/SEND, Covid and/or numerous other changes.
4. Some observers appear to construct a false Scotland–England natural experiment.
England isn't Scotland with one variable (curriculum) changed. Both education systems differ in governance, academisation, inspection, accountability, examinations, league tables, selection, exclusions, SEND/ASN practice, school structures and so much more. Just by calling curriculum the decisive difference doesn't necessarily demonstrate that it is?
One particularly important given is that some observers/ commentators treat England's PISA result as though it were an unproblematic comparator. It might appear this is not the case? England's own 2025 analysis found participation was significantly related to school attainment: pupils in the lowest-attaining schools were less likely to participate and those in the highest-attaining schools more likely to participate. Pupils in selective schools were also more likely to participate. England initially achieved only 78% participation among originally sampled schools and used replacement schools to reach 90%.
To be precise, I am not arguing, as some have, that this proves deliberate English “gaming” of the processes. But surely this means we cannot simply treat the achieved English sample as an unbiased experimental control for Scotland.
And it would appear this isn't merely a theoretical concern. My understanding is that the OECD's investigation of the 2022 English/UK sample found residual upward performance bias associated with non-response- approximately 0.07 SD in reading and 0.09 SD in mathematics.
5. Too many observers caricature CfE into a simplistic trope of “skills instead of knowledge”. Anyone at the chalkface of our system knows this is simply not an accurate portrayal of Scottish secondary education. Secondary S1/2 CfE contains significant disciplinary content, and, as you know, our Senior Phase involves subject-specific National Qualifications with prescribed content and national assessment. For example, a 15-year-old Scottish pupil taking National 5 Chemistry, Maths or History is not sitting in a classroom acquiring generic “skills” instead of knowledge. If observers/ critics claim that specific knowledge deficit caused the PISA decline, surely they need to identify the missing knowledge, demonstrate that CfE 'implementation' removed or reduced it, and evidence that this deficit explains the PISA performance difference.
6. Some observers appear to misunderstand the very instrument they are using to make their argument. PISA (deliberately) is not a straightforward test of mastery of a national curriculum, rather it assesses/ measures assesses pupils' ability to apply knowledge and skills, reason and solve problems in contexts beyond simple reproduction of taught curricular material. A low PISA score does not diagnose in itself “insufficient knowledge”. Some commentators appear to have inserted this 'diagnosis' themselves.
7. We need to be careful as to how we explain PISA's rankings. "England sixth; Scotland 21st” sounds dramatic but may deliver spurious precision? I think PISA scores are estimates with confidence intervals; many countries separated in any ordinal 'league table' are not statistically distinguishable. The meaningful questions concern scores, distributions, effect sizes, trends and uncertainty- not whether a country happens to 'occupy' box 16 or 21 in a table someone has constructed.
8. Some observers/ commentators are a bit too keen to turn what is in essence an attainment-distribution observation into political rhetoric. If Scotland's socio-economic attainment gap has narrowed partly because higher-SES pupils' performance has fallen then this is clearly genuinely important and deserves investigation; but calling it, as some have, an “attack on the middle class” is not on any way an 'empirical' conclusion. PISA on its own surely cannot establish either governmental intention or any causal mechanisms responsible.
My own evidence is that Scottish secondary schools deliver a subject (knowledge)-based, schema focused, teacher-led/ student-centred approach to learning (with obvious inherent inconsistencies- as revealed in ongoing HMIE reports).The 'knowledge debate' is one that exists in our education quangoland and not at the chalkface. Indeed, even after all this time it is difficult to clarify exactly what CFE actually 'means'. Stating that CfE is the cause of our concerns is lazy at best if we do not define what CfE actually represents.
There are important issues raised by the latest PISA data, but my fear is we return to a short term cycle of blame (often with an anti-Scottish, anti- comprehensive bias) and lack of serious analysis over what needs to be done to identify improvements needed. Yes we MAY need to look at our delivery of reading/ literacy skills in primary; yes we MAY need to examine in more detail efficacy of teacher pedagogy; yes we MAY need to review the present curriculum framework design proposals away from the project- based learning trend that appears to be emerging . . .
'A plague on all your houses'. My worry is that, based one one narrow set of 'problematic' data, confidence of our profession takes yet another media hammering from the tit-for-tat 'dialogue' across groups who have their own self-interested 'skin in the education system game' or have their own ideological agendas, but are too far removed from the realities of classroom delivery to have any impact on pupil attainment.
Sorry for the detail, but needed to 'vent' :-) . . .
Sent from my Galaxy
Sent from my Galaxy
Yes it’s ridiculous
Food for thought.
Thanks
I found this very interesting Jackie wonder if Prof John Watson will read it👍👏👏👏👏👏👏
Interesting. Thank you for sharing
Copium