Who makes it into PISA in Turkey?

UJ Curriculum Survey 2017 – https://goo.gl/forms/2sgaWpGKpUKBk1td2 

Below is a quick summary of what I was working on at the OECD in Paris last year. The full paper is now available online here:

2Of the OECD countries that participate in the Programme for International Student Assessment (PISA), Turkey has one of the lowest levels of performance and the highest rates of improvement in PISA scores. However, most analyses have traditionally ignored one vital question: what percentage of 15 year olds in Turkey are eligible for the PISA sample in each wave of PISA? A new OECD Working Paper focuses on this specific question and sheds new light on the performance of Turkey between 2003 and 2012. It shows that the percentage of students that were eligible for PISA in Turkey between 2003 and 2012 nearly doubled from 36% to 68% (using OECD indicators) or from 45% to 80% (using household survey data). This is summarised in Figure 1 below which provides information from the Turkish Demographic and Health Survey (DHS) of 2003, 2009 and 2012.

Figure 1: The educational status and PISA-eligibility of 15-16 year olds in Turkey in DHS 2003, DHS 2008 and DHS 2013

1

While PISA aims to assess a nationally representative sample of 15 year olds, because PISA is a school-based survey, in reality it excludes all students that are no longer in school (due to drop out or non-enrolment). PISA also only samples 15 year olds if they are in Grade 7 or higher. So PISA is really a nationally representative sample of 15 year old students that are still enrolled in school and are currently in Grade 7 or higher. This might sound like a trivial technicality, and in most wealthy OECD countries like Germany or Japan it is. But some OECD countries (like Turkey and Mexico) and many partner countries (like Vietnam and Indonesia) have high levels of student dropout and delay leading to low levels of PISA sample coverage. As an aside, the new PISA-for-Development initiative aims to also survey out-of-school 15 year olds in some of the developing countries participating in that programme.

Since the beginning of PISA, the OECD has reported the percentage of 15 year olds that are actually eligible for PISA, what is called “Coverage Index 3*.” This statistic is calculated using census and enrolment data in each country and is provided in the overall PISA Reports and Technical Reports for all participating countries. For example, in Turkey in PISA 2003, only 36% of 15 year olds were eligible for PISA. That is to say that PISA 2003 in Turkey is only representative of 36% of the country’s 15 year olds. By comparison, the figure in Germany in 2003 was 93%. Table 1 below provides the Coverage Index 3 rates for a selected group of PISA countries with low levels of sample coverage (Germany and Canada are included as reference countries). From this we can see that a number of partner countries have very low levels of sample coverage, including Costa Rica, Indonesia, Peru and Vietnam, but also that some OECD countries (such as Brazil, Mexico and Turkey) have low levels of sample coverage.

Table 1: The percentage of the total 15 year old population covered by the PISA sampling frame (Coverage Index 3) in selected countries

Screen Shot 2017-04-18 at 7.57.17 AM

Given these low levels of sample coverage in some countries, it is reasonable to ask: how would the results change if we included PISA-ineligible 15-year-olds in our calculations? This is the focus of a new working paper published this week, titled “Underestimating Progress and Inequality in Turkey (2003-2012): Using PISA and DHS to form a complete picture of access and quality”. The method and results are briefly summarised below.

The paper applies a new method developed by Spaull & Taylor (2015) which aims to combine statistics on the learning outcomes of 15 year olds that are still in school (using PISA) with data on the number and type of 15 year olds that are not in school (using household-survey data). By assuming that PISA-ineligible students would not have reached PISA Level 2 in Reading and Mathematics – a relatively conservative assumption – we can calculate the percentage of the total population of 15 year olds that reach Level 2 in PISA, rather than only the percentage of those that are still in school. Figure 2 and Figure 3 below provide these breakdowns for Turkey in PISA 2003 and PISA 2012 respectively, and also by gender and socioeconomic subgroups. (Note: ‘Poor40’ means the poorest 40% of 15-16 year olds, and ‘Poor40F’ means poorest 40% of 15-16 year olds that are also female).

Figure 2 Access to Literacy (Level 2) in Turkey 2003 (PISA 2003 and DHS 2003)

Screen Shot 2017-04-18 at 7.57.47 AM

Figure 3 Access to Literacy (Level 2) in Turkey 2012 (PISA 2012 and DHS 2013)

Screen Shot 2017-04-18 at 7.57.59 AM

Note the results above will be different from those found in PISA reports because these results include PISA ineligible 15-16 year olds in the calculations.

The 7 main findings from the above are as follows:

  1. There have been significant increases in PISA sample eligibility over time from around 45% of 15-16 year olds in 2003 to 80% in 2013.
  2. If we account for the growth in PISA eligible 15-16 year olds, the improvement in the percentage of 15-16 year olds acquiring Level 2 is between 2 times larger (for Mathematics) and 5 times larger (in Reading) than previously thought.
  3. Severe delays in grade progression in Turkey have been virtually eliminated and dropout has declined by 42% over this period.
  4. In 2003 the richest group were twice as likely to be eligible for the PISA sample than the poorest group.
  5. Although both boys and girls benefitted over the period, girls benefitted the most.
  6. The gap in access-to-literacy and access-to-numeracy between rich and poor has not changed and is larger than previously thought.
  7. 15-16 year olds in the East region of Turkey are less likely to be eligible for the PISA sample than 15-16 year olds in other regions.

Whether one chooses to use the Coverage Index 3 figures reported by the OECD itself, or those calculated from various DHS surveys, the conclusion is the same: there are large and changing proportions of Turkish students that do not make it into the PISA sampling frame and this has a substantial effect on the validity of inter-country and inter-temporal comparisons. This analysis shows that the gains in Turkey between 2003 and 2012 have actually been even more impressive than if one only looked at PISA data. This method could also usefully be applied to other middle-income and developing countries with high and changing percentages of PISA-eligible 15 and 16 year olds.

References

OECD. (2016). PISA for Development: Benefits for participating countries. PISA for Development Brief (Vol. 2). Paris.

Spaull, N., and Taylor, S., (2015). Access to what? Creating a composite measure of educational quantity and educational quality for 11 African countries. Comparative Education Review. Vol. 58, No. 1.

*There is an error in the Coverage Index 3 values provided in the PISA 2003 Report. The correct Coverage Index 3 values for 2003 can be found in the PISA Technical Report for 2003. See the Working Paper for a full discussion.

The Godfather Speaks

godfYesterday RESEP had our first internal education workshop of the year and it was a great success. One of the presentations was by Martin Gustafsson on various trends and shocks in the  education system. I found it to be especially important and interesting and asked if I could include the presentation in blog format, which Martin kindly agreed to (see below). For those that don’t know Martin he is easily one of the top 3 most knowledgeable people about the SA education system.

The 2 main points that Martin makes are:

  1. There is irrefutable evidence that there has been real progress in learning outcomes in SA over the last decade,
  2. That there is a very strange demographic trend in SA (confirmed by 2 independent data sources) that there was a large increase in births in 2004 and 2005 (in the order of 10%) which is extremely peculiar. Note this has significant implications for resource allocation, class sizes etc. This was discussed in the DBE 2016 Sector Review (see excerpt below), but hasn’t been given nearly enough attention.

123

Regarding #2, from discussions with Martin, who has been involved in reviewing the 2013 SACMEQ results, it seems clear that the 2007 to 2013 SACMEQ trend has indeed been positive for South Africa, and about as large as one could expect (and roughly in line with what has been seen in TIMSS). But these positive trends are smaller than the extremely large improvements, which I’ve argued are implausible, seen in earlier preliminary SACMEQ results presented to the Parliamentary Portfolio Committee on Basic Education in 2016. What the sector urgently needs is the finalising of the 2013 SACMEQ process and the release of official learner results for all SACMEQ countries.

The trends seen in #3 below are from this 2016 paper of Martin’s.

456

“At the lower primary level, what has clearly occurred is the entrance of a ‘wave’ of larger birth cohorts. In 2011, Grade 1 enrolments increased by 5.2% relative to 2010. In 2012, Grade 2 enrolments increased by 7.0%. By 2015, the increase had reached Grade 5, with enrolments in this grade increasing by 5.2% relative to 2014. As seen in Table 2 in the appendix, substantial increases were seen across all provinces, with the exception of Eastern Cape, which in fact saw a decline . The increases were strongest in Gauteng and Western Cape, almost certainly because these two provinces experienced a combination of two factors, both the demographic change in terms of larger birth cohorts, plus migration into these provinces. By 2015, grades 1 to 7 enrolments had reached 7,1 million in public ordinary schools, the highest figure since 2007 (when the enrolment total was also around 7.1 million). Moreover, total enrolment in just grades 1 to 5 was higher than it had been in any year since 2002. Projections indicate that the increases will continue to be felt in grades beyond Grade 5 in 2016 and beyond. ” Page 9 of this 2016 DBE Sector Report.

7

*Note in the graph s above (top left) the steady rise in births from 1982 to 1998 are simply because of better data-capturing rather than increased births. It stabilises from about 2000.

8910

For those that haven’t read the DBE 2016 Sector Review I would strongly recommend doing so!

Links on 21st C teaching…

21

I’ve been giving a few presentations on 21st Century Skills and thought it’d be helpful to provide some of the links I like on this topic. If you have any you’d like me to add please do include them in the comments.

Helpful links and reports:

wef

maker

Useful websites:

casel

Some videos I like:

Esther Wojcicki – Empowering Students

Screen Shot 2017-03-20 at 9.26.01 AM

Linda Darling-Hammond – New Learning for a Rapidly Changing World

Screen Shot 2017-03-20 at 9.26.47 AM

And what does the future look like? I like a few of the new WEF videos on The Fourth Industrial Revolution 

Screen Shot 2017-03-20 at 9.57.30 AM

and The New Vision for Education and

Screen Shot 2017-03-20 at 9.59.42 AM

“Matric really does start in Gr1” – my M&G/Teacher article on TIMSS 2015

screen-shot-2017-01-31-at-7-05-23-am

(The article below first appeared in The Teacher magazine in January, and in the Mail & Guardian on the 27th of January 2017)

The month of January is an interesting one in the South African calendar. It seems that this is the one time in the year when everyone cares about education, or more accurately, cares about matric. And in some senses there is a good reason for this; matric is the gateway to higher education. If we look at 25-35 year olds in the 2011 Census data we see that for those with less than a matric, 47% were unemployed (using the broad definition), compared to 33% among those whose highest qualification was a matric and only 8% for those with a bachelors degree. While matric used to be the big distinguisher between the haves and have-nots, increasingly the difference is between those with some kind of tertiary qualification and everyone else. And in 2016, as usual, mathematics proved to be a tough gateway subject for those who want to study further. Only one in three learners who wrote Mathematics in 2016 got 40% or more in the subject, with a recent study showing that only about 15,000 matrics achieved 70% or more in mathematics. That’s about 1,5% of students who start school in Grade 1.

As anyone familiar with maths will tell you, the subject is a hierarchical one that builds upon itself. You need to understand multiplication and division before you can understand fractions or rate and proportion. So where do the wheels come off? Is it just before matric, perhaps in Grade 9 or 10? Or even earlier in Grade 4 or 5? To shed some light on this question we can turn to some recent mathematics research published by the Human Sciences Research Council (HSRC) at the end of last year. Every four years South Africa participates in an international study of mathematics achievement called the Trends in International Mathematics and Science Study or TIMSS. We test a nationally-representative sample of Grade 5 and Grade 9 learners on a standardised international test, which is the same test that is written by thousands of other students in other countries and allows us to compare ourselves to these other countries. (Although almost all the countries test their Grade 4 and Grade 8 students on these tests). The most recent round of tests was done in 2015.

The results showed that only 34% of our Grade 9 learners could do basic mathematics, i.e. could reach the lowest international benchmark. That is to say that 66% of our learners could not do basic computations or match tables to bar graphs or read a simple line graph. They had not acquired a basic understanding about whole numbers, decimals, operations or basic graphs. However there was one piece of encouraging information that emerged from the study and that was that the number of students that could do basic mathematics increased from 24% in 2011 to 34% in 2015 in South Africa, one of the fastest improvements seen internationally.

But if 66% of our Grade 9 learners can’t do basic maths, when did these gaps emerge? In 2015, for the first time in our history, South Africa also participated in the primary-school level of TIMSS that usually tests Grade 4 learners – we tested our Grade 5 learners. It showed that 61% of Grade 5 learners could not reach the Low International Benchmark. Learners who do not reach the low international benchmark at the primary level cannot add and subtract three or four digit whole numbers like 218 and 191. They do not recognise familiar geometric shapes or parallel and perpendicular lines. They cannot read and complete simple bar graphs and tables. For example, according to our curriculum, multiplying a 3-digit number by a 1-digit number is meant to be covered in Term 1 of Grade 4 yet only 41% of our Grade 5 learners could calculate “512 x 3 =____”, which was one of the questions in 2015. So almost 60% of our Grade 5 learners are already significantly behind the curriculum in 2015.

The conclusion that the root of our problems is not in high school but rather much earlier, in the lower grades of primary school is not a new finding – there are many studies showing this from at least 1999. In a report published nearly 10 years ago, one South African education researcher (Dr Eric Scholar) analysed the mathematics achievement in 154 schools across the country. He concluded that the low levels of achievement we see in the higher grades are rooted in weak foundations in primary school. To quote his exact conclusion:

‘‘The fundamental cause of poor learner performance across our education system is a failure to extend the ability of learners from counting to true calculating in their primary schooling. All more complex mathematics depends, in the first instance, on an instinctive understanding of place value within the base-10 number system, combined with an ability to readily perform basic calculations and see numeric relationships … Learners are routinely promoted from one Grade to the next without having mastered the content and foundational competences of preceding Grades, resulting in a large cognitive backlog that progressively inhibits the acquisition of more complex competencies. The consequence is that every class has become, in effect, a ‘multi-Grade’ class in which there is a very large range of learner abilities and this makes it very difficult, or even impossible, to consistently teach to the required assessment standards for any particular Grade. Mathematics, however, is a hierarchical subject in which the development of increasingly complex cognitive abilities at each succeeding level is dependent on the progressive and cumulative mastery of its conceptual frameworks, starting with the absolutely fundamental basics of place value (the base-10 number system) and the four operations (calculation)’’

While as a country we continue to obsess about the matric pass rate, the research is really quite clear. The majority of our young people are acaquiring learning deficits early on in primary school and then carrying these with them as they move through school. As they are promoted into higher grades there is a decoupling between what learners know and can do and what the curriculum expects from them. We need to acknowledge that matric starts in Grade 1 (and even earlier), and that it really is possible to improve primary schooling if that is where we focus most of our time, energy and resources.

DBE needs you (2 vacancies)

we-need-you

My good friend and colleague Dr Stephen Taylor (who is the Director of Research in the National Department of Basic Education (DBE) in Pretoria) is looking for 2 people to join his team in the Research Monitoring and Evaluation unit. The full job advertisement is here and I’ve included screenshots of the relevant position below. I’ve done a Q&A with Stephen (and Martin Gustafsson who is also in DBE) to give you a flavour of the kind of people they are. I would whole-heartedly recommend Stephen as a boss and a person 🙂 they are creating a really impactful team (Janeli Kotze and Mpumi Mohohlwane have also recently been appointed there full-time). If you are looking for an excellent team, to have an impact in education and have good quantitative skills – apply! (Deadline 17 Feb).

screen-shot-2017-01-30-at-8-01-34-am

screen-shot-2017-01-30-at-8-01-43-am

PS I should totally get a finders fee for this kind of thing 🙂

Yale Young African Scholars Program 2017 (high-school students)

yyas-2017-flyer

If you know of any outstanding high-school students across the continent, this looks like a great opportunity to help prepare them for the US university admission and financial aid processes. Please forward it on to them. Deadline is the 31st of January 2017.

See excerpt from the email below

In 2016 YYAS brought 300 of Africa’s most talented high school students to programs in Rwanda, Ghana and Zimbabwe where they engaged in the sorts of robust intellectual exchanges that are crucial to understanding Africa’s most pressing challenges and exciting opportunities. With participants from over 20 countries across the continent, the students also gained exposure to the ideas and perspectives of a very diverse group of peers and took part in leadership skills development activities. Critically, however, the main focus of the YYAS program is to introduce the participants to U.S. universities’ admission and financial aid processes. Workshops led by Yale students as well as university admission representatives equipped YYAS participants with a thorough understanding of these processes, and we also offered standardized test preparation sessions plus ongoing mentorship that will see each student through his or her university application cycle.

I am very proud to be joining this exciting program and for the opportunity to help YYAS further its reach and scale across the continent. And I’m thrilled to announce that the 2017 YYAS program application is open! I hope you will share our promotional materials (see attachments) with secondary school students or organizations that work with secondary school students in Africa. Moreover, we do anticipate that many of our program’s alumni will apply to your university or programs in the near future. Please watch out for YYAS on their applications! We are certain that the academic standing and caliber of our alumni will make them excellent members of your community and hope that you will see YYAS as a stepping stone to great success in their university studies.

 

My M&G article on Universities in 2017

screen-shot-2017-01-08-at-1-10-29-pm

(The above article was published in the Mail & Guardian on the 6th of January 2016 and is available in PDF here).

Students contest the status quo 

Over the last two years, universities in South Africa have become increasingly contested spaces. Student movements like RhodesMustFall and FeesMustFall have categorically rejected the status quo as unacceptable and are working to reorder not only the principles that govern universities, but ultimately the principles that govern the country. Of course the first order of business is challenging our current assumptions about who should go to university, what it should look like, and who should pay for it. And on all three fronts they have been phenomenally successful. It is really quite remarkable that a loose group of students who lack a political mandate, who have not been elected by anyone, and have virtually no resources have managed to achieve so much so quickly. They have brought whole universities to their knees and prompted the creation of a Presidential Task Team. Most significantly they garnered enough support to essentially force the government to allocate an additional R17 billion to higher education in the Medium Term Budget.

About 200 years ago, Napoleon Bonaparte famously quipped that a revolution is simply an idea which has found its bayonets. In the context of the various student movements I think it’s worthwhile to try and identify the underlying idea, its animating principle. As the student movements assemble and reassemble under different names (seemingly quite effortlessly), I think there is a leitmotif running through all of them; the unfinished business of 1994. There is a generation of young Black South Africans who feel that the terms of the negotiated settlement were unjust and let White South Africans off the hook. Dr Amos Wilson. a theoretical psychologist and social theorist, makes the logic behind this position explicit in the following quote:

“Justice requires not only the ceasing and desisting of injustice but also requires either punishment or reparation for injuries and damages inflicted for prior wrongdoing. The essence of justice is the redistribution of gains earned through the perpetration of injustice. If restitution is not made and reparations not instituted to compensate for prior injustices, those injustices are in effect rewarded. And the benefits such rewards conferred on the perpetrators of injustice will continue to “draw interest,” to be reinvested, and to be passed on to their children, who will use their inherited advantages to continue to exploit the children of the victims of the injustice of their ancestors. Consequently, injustice and inequality will be maintained across generations as will their deleterious social, economic, and political outcomes.”

Thinking that the various incarnations of the student movements are primarily about universities is a mistake. RhodesMustfall was not about a statue; it was about reclamation and power and history. Similarly, the challenge today is not only about who should pay fees, but who should own the land. The discontent and anger about the ‘pay-to-play’ market system that we have – where only those who can pay for quality get it – is as much about private hospitals and Model-C schools as it is about universities. The true contested space at our universities at the moment is really about the principles that currently order our society and reimagining different ones.

Fighting for a different future

There are students in South Africa today who look at our country and refuse to accept that the way we are currently doing things is the only way they can be done. How is it that in a country with considerable wealth and resources that we still have 10 million people living on less than R10 a day? Whenever I land at Cape Town International Airport and get an aerial view of Khayelitsha, I think to myself “How the heck can we, as a country, not find a dignified solution to housing for the poor?” In Cape Town we have 400,000 people living in shacks a mere 40-minute drive from the house that sold for R290-million in Bantry Bay. We have decadent opulence living next to extreme poverty. It’s not right.

And so we come back to the contested space at universities where people have different ideas about how we get from where we are to a better future. Students associated with Black-First-Land-First argue for land expropriation without compensation. The Nobel Laureate Thomas Piketty motivates for much steeper wealth and inheritance taxes to level the playing field. The Wits SRC has proposed a once-off ‘apartheid windfall’ tax on “companies that benefited unfairly by abusing state resources” under apartheid.

But since the current discussions at universities are still centred on fees and access to university, let’s start there and think about what 2017 might hold for universities, and put some numbers on the table. Personally I think we will actually find a sustainable solution to student financing at universities, possibly even in 2017. Sizwe Nxasana – the head of the Presidential Task Team – has developed a highly sophisticated and workable model of student funding called the Ikusasa Student Financial Aid Programme (ISFAP) that is being trialled at seven universities this year, focussing on students studying medicine, engineering and accounting. This is essentially a public-private partnership which aims to “significantly increase the funding and resources which are made available to support students from working class families to graduate and find employment by leveraging private sector funding.” One can think of it as a three-tier model with the poorest students being fully funded with grants and the missing-middle with a combination of grants and income-contingent loans (to be repaid only if the recipient does graduate and earns above a certain amount). Then finally, those at the top that can pay fees do pay fees. While it isn’t free education for everyone – and the vanguard may therefore not accept it – if implemented properly it has a good shot at ensuring that no student is excluded from university on financial grounds. That would be a significant achievement.

Thankfully, many in the sector are now realising that needs-blind allocations to higher education – where all students are equally subsidised – are socially regressive and anti-poor. This is largely because the children of the wealthy attend fee-charging schools that give them a much better shot at qualifying for university than the children of the poor. We know that less than 1 in 10 children from the poorest 70% of households qualify to go to university compared to 1 in 2 or 3 children (40%) among the wealthiest 10% of households. And because of this, if one allocated an additional R10bn to higher-education in a blanket fashion, then about R6,8bn (68%) will end up benefitting the wealthiest 20% of South African households because it is their children who are disproportionately at university (according to two fiscal incidence studies). A recent study showed that 60% of students that qualified for university came from the 30% of high schools that charged fees. What is the point of raising revenue by additional taxes on the richest 20% only to give two thirds of that money straight back to them in the form of indirect subsidies to their children?

So if we agree that the rich should not be subsidised (usually defined as those in households with annual income of more than R600,000), how many students would need funding? Professor Servaas van der Berg’s analysis of household surveys has shown that about 60% of the current university-going population would be eligible for funding. (This assumes that income is under-captured in surveys by about 30% . Importantly, this would cover 73% of Black African university students and 30% of White university students.

While ending financial exclusion at university won’t solve the thornier issues in South Africa – about land, inequality, restitution, primary education, unemployment – it would serve as a powerful and invigorating example that things really can be different to what they are now. It would be poetic if the start of a successful campaign for a different South Africa could trace its origins to the toppling of a statue of Cecil John Rhodes.

Dr Nic Spaull is an education researcher at the Research on Socioeconomic Policy (RESEP) group at Stellenbosch University. He is on Twitter @NicSpaull