Monthly Archives: April 2017

Below is a quick summary of what I was working on at the OECD in Paris last year. The full paper is now available online here:

• Spaull, N. (2017) Who makes it into PISA? Understanding the impact of sample eligibility using Turkey as a case study (PISA 2003- PISA 2012). OECD Education Working Papers, No. 154, OECD Publishing, Paris. DOI: http://dx.doi.org/10.1787/41d175fc-en

Of the OECD countries that participate in the Programme for International Student Assessment (PISA), Turkey has one of the lowest levels of performance and the highest rates of improvement in PISA scores. However, most analyses have traditionally ignored one vital question: what percentage of 15 year olds in Turkey are eligible for the PISA sample in each wave of PISA? A new OECD Working Paper focuses on this specific question and sheds new light on the performance of Turkey between 2003 and 2012. It shows that the percentage of students that were eligible for PISA in Turkey between 2003 and 2012 nearly doubled from 36% to 68% (using OECD indicators) or from 45% to 80% (using household survey data). This is summarised in Figure 1 below which provides information from the Turkish Demographic and Health Survey (DHS) of 2003, 2009 and 2012.

Figure 1: The educational status and PISA-eligibility of 15-16 year olds in Turkey in DHS 2003, DHS 2008 and DHS 2013

While PISA aims to assess a nationally representative sample of 15 year olds, because PISA is a school-based survey, in reality it excludes all students that are no longer in school (due to drop out or non-enrolment). PISA also only samples 15 year olds if they are in Grade 7 or higher. So PISA is really a nationally representative sample of 15 year old students that are still enrolled in school and are currently in Grade 7 or higher. This might sound like a trivial technicality, and in most wealthy OECD countries like Germany or Japan it is. But some OECD countries (like Turkey and Mexico) and many partner countries (like Vietnam and Indonesia) have high levels of student dropout and delay leading to low levels of PISA sample coverage. As an aside, the new PISA-for-Development initiative aims to also survey out-of-school 15 year olds in some of the developing countries participating in that programme.

Since the beginning of PISA, the OECD has reported the percentage of 15 year olds that are actually eligible for PISA, what is called “Coverage Index 3*.” This statistic is calculated using census and enrolment data in each country and is provided in the overall PISA Reports and Technical Reports for all participating countries. For example, in Turkey in PISA 2003, only 36% of 15 year olds were eligible for PISA. That is to say that PISA 2003 in Turkey is only representative of 36% of the country’s 15 year olds. By comparison, the figure in Germany in 2003 was 93%. Table 1 below provides the Coverage Index 3 rates for a selected group of PISA countries with low levels of sample coverage (Germany and Canada are included as reference countries). From this we can see that a number of partner countries have very low levels of sample coverage, including Costa Rica, Indonesia, Peru and Vietnam, but also that some OECD countries (such as Brazil, Mexico and Turkey) have low levels of sample coverage.

Table 1: The percentage of the total 15 year old population covered by the PISA sampling frame (Coverage Index 3) in selected countries

Given these low levels of sample coverage in some countries, it is reasonable to ask: how would the results change if we included PISA-ineligible 15-year-olds in our calculations? This is the focus of a new working paper published this week, titled “Underestimating Progress and Inequality in Turkey (2003-2012): Using PISA and DHS to form a complete picture of access and quality”. The method and results are briefly summarised below.

The paper applies a new method developed by Spaull & Taylor (2015) which aims to combine statistics on the learning outcomes of 15 year olds that are still in school (using PISA) with data on the number and type of 15 year olds that are not in school (using household-survey data). By assuming that PISA-ineligible students would not have reached PISA Level 2 in Reading and Mathematics – a relatively conservative assumption – we can calculate the percentage of the total population of 15 year olds that reach Level 2 in PISA, rather than only the percentage of those that are still in school. Figure 2 and Figure 3 below provide these breakdowns for Turkey in PISA 2003 and PISA 2012 respectively, and also by gender and socioeconomic subgroups. (Note: ‘Poor40’ means the poorest 40% of 15-16 year olds, and ‘Poor40F’ means poorest 40% of 15-16 year olds that are also female).

Figure 2 Access to Literacy (Level 2) in Turkey 2003 (PISA 2003 and DHS 2003)

Figure 3 Access to Literacy (Level 2) in Turkey 2012 (PISA 2012 and DHS 2013)

Note the results above will be different from those found in PISA reports because these results include PISA ineligible 15-16 year olds in the calculations.

The 7 main findings from the above are as follows:

1. There have been significant increases in PISA sample eligibility over time from around 45% of 15-16 year olds in 2003 to 80% in 2013.
2. If we account for the growth in PISA eligible 15-16 year olds, the improvement in the percentage of 15-16 year olds acquiring Level 2 is between 2 times larger (for Mathematics) and 5 times larger (in Reading) than previously thought.
3. Severe delays in grade progression in Turkey have been virtually eliminated and dropout has declined by 42% over this period.
4. In 2003 the richest group were twice as likely to be eligible for the PISA sample than the poorest group.
5. Although both boys and girls benefitted over the period, girls benefitted the most.
6. The gap in access-to-literacy and access-to-numeracy between rich and poor has not changed and is larger than previously thought.
7. 15-16 year olds in the East region of Turkey are less likely to be eligible for the PISA sample than 15-16 year olds in other regions.

Whether one chooses to use the Coverage Index 3 figures reported by the OECD itself, or those calculated from various DHS surveys, the conclusion is the same: there are large and changing proportions of Turkish students that do not make it into the PISA sampling frame and this has a substantial effect on the validity of inter-country and inter-temporal comparisons. This analysis shows that the gains in Turkey between 2003 and 2012 have actually been even more impressive than if one only looked at PISA data. This method could also usefully be applied to other middle-income and developing countries with high and changing percentages of PISA-eligible 15 and 16 year olds.

References

OECD. (2016). PISA for Development: Benefits for participating countries. PISA for Development Brief (Vol. 2). Paris.

Spaull, N., and Taylor, S., (2015). Access to what? Creating a composite measure of educational quantity and educational quality for 11 African countries. Comparative Education Review. Vol. 58, No. 1.

*There is an error in the Coverage Index 3 values provided in the PISA 2003 Report. The correct Coverage Index 3 values for 2003 can be found in the PISA Technical Report for 2003. See the Working Paper for a full discussion.

Yesterday RESEP had our first internal education workshop of the year and it was a great success. One of the presentations was by Martin Gustafsson on various trends and shocks in the  education system. I found it to be especially important and interesting and asked if I could include the presentation in blog format, which Martin kindly agreed to (see below). For those that don’t know Martin he is easily one of the top 3 most knowledgeable people about the SA education system.

The 2 main points that Martin makes are:

1. There is irrefutable evidence that there has been real progress in learning outcomes in SA over the last decade,
2. That there is a very strange demographic trend in SA (confirmed by 2 independent data sources) that there was a large increase in births in 2004 and 2005 (in the order of 10%) which is extremely peculiar. Note this has significant implications for resource allocation, class sizes etc. This was discussed in the DBE 2016 Sector Review (see excerpt below), but hasn’t been given nearly enough attention.

Regarding #2, from discussions with Martin, who has been involved in reviewing the 2013 SACMEQ results, it seems clear that the 2007 to 2013 SACMEQ trend has indeed been positive for South Africa, and about as large as one could expect (and roughly in line with what has been seen in TIMSS). But these positive trends are smaller than the extremely large improvements, which I’ve argued are implausible, seen in earlier preliminary SACMEQ results presented to the Parliamentary Portfolio Committee on Basic Education in 2016. What the sector urgently needs is the finalising of the 2013 SACMEQ process and the release of official learner results for all SACMEQ countries.

The trends seen in #3 below are from this 2016 paper of Martin’s.

“At the lower primary level, what has clearly occurred is the entrance of a ‘wave’ of larger birth cohorts. In 2011, Grade 1 enrolments increased by 5.2% relative to 2010. In 2012, Grade 2 enrolments increased by 7.0%. By 2015, the increase had reached Grade 5, with enrolments in this grade increasing by 5.2% relative to 2014. As seen in Table 2 in the appendix, substantial increases were seen across all provinces, with the exception of Eastern Cape, which in fact saw a decline . The increases were strongest in Gauteng and Western Cape, almost certainly because these two provinces experienced a combination of two factors, both the demographic change in terms of larger birth cohorts, plus migration into these provinces. By 2015, grades 1 to 7 enrolments had reached 7,1 million in public ordinary schools, the highest figure since 2007 (when the enrolment total was also around 7.1 million). Moreover, total enrolment in just grades 1 to 5 was higher than it had been in any year since 2002. Projections indicate that the increases will continue to be felt in grades beyond Grade 5 in 2016 and beyond. ” Page 9 of this 2016 DBE Sector Report.

*Note in the graph s above (top left) the steady rise in births from 1982 to 1998 are simply because of better data-capturing rather than increased births. It stabilises from about 2000.

For those that haven’t read the DBE 2016 Sector Review I would strongly recommend doing so!