Nic Spaull

Below is a quick summary of what I was working on at the OECD in Paris last year. The full paper is now available online here:

• Spaull, N. (2017) Who makes it into PISA? Understanding the impact of sample eligibility using Turkey as a case study (PISA 2003- PISA 2012). OECD Education Working Papers, No. 154, OECD Publishing, Paris. DOI: http://dx.doi.org/10.1787/41d175fc-en

Of the OECD countries that participate in the Programme for International Student Assessment (PISA), Turkey has one of the lowest levels of performance and the highest rates of improvement in PISA scores. However, most analyses have traditionally ignored one vital question: what percentage of 15 year olds in Turkey are eligible for the PISA sample in each wave of PISA? A new OECD Working Paper focuses on this specific question and sheds new light on the performance of Turkey between 2003 and 2012. It shows that the percentage of students that were eligible for PISA in Turkey between 2003 and 2012 nearly doubled from 36% to 68% (using OECD indicators) or from 45% to 80% (using household survey data). This is summarised in Figure 1 below which provides information from the Turkish Demographic and Health Survey (DHS) of 2003, 2009 and 2012.

Figure 1: The educational status and PISA-eligibility of 15-16 year olds in Turkey in DHS 2003, DHS 2008 and DHS 2013

While PISA aims to assess a nationally representative sample of 15 year olds, because PISA is a school-based survey, in reality it excludes all students that are no longer in school (due to drop out or non-enrolment). PISA also only samples 15 year olds if they are in Grade 7 or higher. So PISA is really a nationally representative sample of 15 year old students that are still enrolled in school and are currently in Grade 7 or higher. This might sound like a trivial technicality, and in most wealthy OECD countries like Germany or Japan it is. But some OECD countries (like Turkey and Mexico) and many partner countries (like Vietnam and Indonesia) have high levels of student dropout and delay leading to low levels of PISA sample coverage. As an aside, the new PISA-for-Development initiative aims to also survey out-of-school 15 year olds in some of the developing countries participating in that programme.

Since the beginning of PISA, the OECD has reported the percentage of 15 year olds that are actually eligible for PISA, what is called “Coverage Index 3*.” This statistic is calculated using census and enrolment data in each country and is provided in the overall PISA Reports and Technical Reports for all participating countries. For example, in Turkey in PISA 2003, only 36% of 15 year olds were eligible for PISA. That is to say that PISA 2003 in Turkey is only representative of 36% of the country’s 15 year olds. By comparison, the figure in Germany in 2003 was 93%. Table 1 below provides the Coverage Index 3 rates for a selected group of PISA countries with low levels of sample coverage (Germany and Canada are included as reference countries). From this we can see that a number of partner countries have very low levels of sample coverage, including Costa Rica, Indonesia, Peru and Vietnam, but also that some OECD countries (such as Brazil, Mexico and Turkey) have low levels of sample coverage.

Table 1: The percentage of the total 15 year old population covered by the PISA sampling frame (Coverage Index 3) in selected countries

Given these low levels of sample coverage in some countries, it is reasonable to ask: how would the results change if we included PISA-ineligible 15-year-olds in our calculations? This is the focus of a new working paper published this week, titled “Underestimating Progress and Inequality in Turkey (2003-2012): Using PISA and DHS to form a complete picture of access and quality”. The method and results are briefly summarised below.

The paper applies a new method developed by Spaull & Taylor (2015) which aims to combine statistics on the learning outcomes of 15 year olds that are still in school (using PISA) with data on the number and type of 15 year olds that are not in school (using household-survey data). By assuming that PISA-ineligible students would not have reached PISA Level 2 in Reading and Mathematics – a relatively conservative assumption – we can calculate the percentage of the total population of 15 year olds that reach Level 2 in PISA, rather than only the percentage of those that are still in school. Figure 2 and Figure 3 below provide these breakdowns for Turkey in PISA 2003 and PISA 2012 respectively, and also by gender and socioeconomic subgroups. (Note: ‘Poor40’ means the poorest 40% of 15-16 year olds, and ‘Poor40F’ means poorest 40% of 15-16 year olds that are also female).

Figure 2 Access to Literacy (Level 2) in Turkey 2003 (PISA 2003 and DHS 2003)

Figure 3 Access to Literacy (Level 2) in Turkey 2012 (PISA 2012 and DHS 2013)

Note the results above will be different from those found in PISA reports because these results include PISA ineligible 15-16 year olds in the calculations.

The 7 main findings from the above are as follows:

1. There have been significant increases in PISA sample eligibility over time from around 45% of 15-16 year olds in 2003 to 80% in 2013.
2. If we account for the growth in PISA eligible 15-16 year olds, the improvement in the percentage of 15-16 year olds acquiring Level 2 is between 2 times larger (for Mathematics) and 5 times larger (in Reading) than previously thought.
3. Severe delays in grade progression in Turkey have been virtually eliminated and dropout has declined by 42% over this period.
4. In 2003 the richest group were twice as likely to be eligible for the PISA sample than the poorest group.
5. Although both boys and girls benefitted over the period, girls benefitted the most.
6. The gap in access-to-literacy and access-to-numeracy between rich and poor has not changed and is larger than previously thought.
7. 15-16 year olds in the East region of Turkey are less likely to be eligible for the PISA sample than 15-16 year olds in other regions.

Whether one chooses to use the Coverage Index 3 figures reported by the OECD itself, or those calculated from various DHS surveys, the conclusion is the same: there are large and changing proportions of Turkish students that do not make it into the PISA sampling frame and this has a substantial effect on the validity of inter-country and inter-temporal comparisons. This analysis shows that the gains in Turkey between 2003 and 2012 have actually been even more impressive than if one only looked at PISA data. This method could also usefully be applied to other middle-income and developing countries with high and changing percentages of PISA-eligible 15 and 16 year olds.

References

OECD. (2016). PISA for Development: Benefits for participating countries. PISA for Development Brief (Vol. 2). Paris.

Spaull, N., and Taylor, S., (2015). Access to what? Creating a composite measure of educational quantity and educational quality for 11 African countries. Comparative Education Review. Vol. 58, No. 1.

*There is an error in the Coverage Index 3 values provided in the PISA 2003 Report. The correct Coverage Index 3 values for 2003 can be found in the PISA Technical Report for 2003. See the Working Paper for a full discussion.

Yesterday RESEP had our first internal education workshop of the year and it was a great success. One of the presentations was by Martin Gustafsson on various trends and shocks in the  education system. I found it to be especially important and interesting and asked if I could include the presentation in blog format, which Martin kindly agreed to (see below). For those that don’t know Martin he is easily one of the top 3 most knowledgeable people about the SA education system.

The 2 main points that Martin makes are:

1. There is irrefutable evidence that there has been real progress in learning outcomes in SA over the last decade,
2. That there is a very strange demographic trend in SA (confirmed by 2 independent data sources) that there was a large increase in births in 2004 and 2005 (in the order of 10%) which is extremely peculiar. Note this has significant implications for resource allocation, class sizes etc. This was discussed in the DBE 2016 Sector Review (see excerpt below), but hasn’t been given nearly enough attention.

Regarding #2, from discussions with Martin, who has been involved in reviewing the 2013 SACMEQ results, it seems clear that the 2007 to 2013 SACMEQ trend has indeed been positive for South Africa, and about as large as one could expect (and roughly in line with what has been seen in TIMSS). But these positive trends are smaller than the extremely large improvements, which I’ve argued are implausible, seen in earlier preliminary SACMEQ results presented to the Parliamentary Portfolio Committee on Basic Education in 2016. What the sector urgently needs is the finalising of the 2013 SACMEQ process and the release of official learner results for all SACMEQ countries.

The trends seen in #3 below are from this 2016 paper of Martin’s.

“At the lower primary level, what has clearly occurred is the entrance of a ‘wave’ of larger birth cohorts. In 2011, Grade 1 enrolments increased by 5.2% relative to 2010. In 2012, Grade 2 enrolments increased by 7.0%. By 2015, the increase had reached Grade 5, with enrolments in this grade increasing by 5.2% relative to 2014. As seen in Table 2 in the appendix, substantial increases were seen across all provinces, with the exception of Eastern Cape, which in fact saw a decline . The increases were strongest in Gauteng and Western Cape, almost certainly because these two provinces experienced a combination of two factors, both the demographic change in terms of larger birth cohorts, plus migration into these provinces. By 2015, grades 1 to 7 enrolments had reached 7,1 million in public ordinary schools, the highest figure since 2007 (when the enrolment total was also around 7.1 million). Moreover, total enrolment in just grades 1 to 5 was higher than it had been in any year since 2002. Projections indicate that the increases will continue to be felt in grades beyond Grade 5 in 2016 and beyond. ” Page 9 of this 2016 DBE Sector Report.

*Note in the graph s above (top left) the steady rise in births from 1982 to 1998 are simply because of better data-capturing rather than increased births. It stabilises from about 2000.

For those that haven’t read the DBE 2016 Sector Review I would strongly recommend doing so!

I have been meaning to blog about some new research on access to higher education that was published earlier this week: “Higher Education Access and Outcomes for the 2008 Matric Cohort” (Van Broekhuizen, Van der Berg & Hofmeyr, 2016). I will only highlight some of the key points from the 122-page Working Paper which is really worth reading in its entirety. Essentially the researchers used the matric data from 2008 and followed these students (using their ID numbers) into the higher education system using data from the Higher Education Management Information System (HEMIS). Perhaps the most striking feature is that of the 100 students that started school, only 12 ever access university (9 immediately after matric and 3 later), 6 get some kind of qualification within 6 years and only 4 get a degree within 6 years.

Secondly, that matrics that attend quintile 5 schools (almost all of which charge fees) are four times as likely to access university than those from the poorest 60% of schools (quintiles 1-3), all of which are no-fee schools. However, it’s encouraging to note that of those quintile 1-3 student that do qualify with a bachelor’s pass, more than 63-68% do actually access university, compared to 70% among quintile 5 students.

Much of the paper points to that fact that unequal access to university is rooted in a highly unequal schooling system where access to high-quality schooling largely depends on a family’s ability to pay school fees. If one looks at the cumulative matric average achievement by race one still finds enormous differentials. While 60% of White matric students achieved 60% or more in matric, only 5% of Black African matrics score at or above 60%. And this is only among the students that actually made it to matric which is only slightly more than half the cohort (see this paper).

The last piece of their research that I want to highlight is that the student intake at different universities is vastly different. If one looks at the matric marks of the typical student entering UCT, Stellenbosch, UP or Wits one can see below that they scored 70% or above on average. This is in stark contrast to those students entering TUT, Fort Hare, Uni-Zulu, Walter Sisulu, UWC etc., all of whom have incoming students whose average matric mark is less than 60%. At the Central University of Technology (CUT, in Free State) the average entrant scored 50% in matric.

At the beginning of last year Professor Servaas van der Berg gave a Brown-Bag Lunch Seminar at Stellenbosch University  on “The Distributional Implications of Student Fees.” I include some notable excerpts and graphs below:

“Education has a number of properties which make the analysis of the demand for it both interesting and complex. … (Education is) …a consumption good and a capital good, i.e., although much of the expenditure is justified in terms of the effects on the individual’s income in the future, many of the activities of educational institutions are primarily justifiable in terms of their immediate consumption benefits. Moreover, education affects individuals’ future incomes.” – (Stiglitz 1974: 349)

Perhaps most striking are Van der Berg’s estimates of who actually makes it to university and where they come from in the income distribution. According to these estimates, there are more students attending university from the richest 10% of the income distribution (Decile 10) than from the poorest 80% of the income distribution (Deciles 1-8 combined).

Last month Nico Cloete (from CHET) gave a lecture at SALDRU (UCT) titled: “University Fees in SA: A Story from Evidence.” I include some relevant slides from his presentation:

Anyone who wants to contribute to the debate about university fees needs to grapple with the realities presented by these three papers/presentations. At the end of the day we need to be able to answer the question of where the money will come from. A Graduate tax? Debt? The Education or Health budgets?

The most reasonable (and probably workable) solution that I have heard is that proposed by Prof Van der Berg who suggests that we should use the existing financial services infrastructure (banks) who could provide government-backed grant-loans (my terminology not SVDB’s) to students that qualify for university. It would be a grant that converts into a loan if a student successfully completes their degree and starts earning a decent income. It would still require a huge amount of government finance to provide the surety to banks for students who come from households that earn less than R500,000 (or some threshold). But, unlike with totally ‘free’ education, the students that do successfully complete their degrees would ‘pay-it-forward’ and contribute to the fund used to finance future students. 

Also, as a side-issue, the Fees Commission needs to get a fast-tracked timetable and told to release at least a preliminary report and recommendations before the end of the year. We cannot wait until June next year. The political hot-potato would have been passed along one too many times from VCs to DHET to Treasury and eventually it will just explode. A stitch in time saves nine.

(If you have any additional research suggestions please send me an email and I’ll include them in this post)

Additional inputs from readers: 

“Between the Devil and Deep Blue Sea? The Financing of Higher Education” 3×3 article by Philippe Burger – Sept 2016 (Thanks Marisa!).

Abstract: “Higher-than-inflation increases in student fees since 2009 often are blamed on declining government subsidies to universities. This is not entirely correct, if one considers real per-student subsidies. Fee increases resulted mainly from cost pressures faced by universities due to growing student numbers and a weakening rand. These pressures will not disappear. Eliminating government wastage is not a durable solution and difficult choices cannot be avoided. So, who should pay for increasing costs, students or government – or which combination of these?”

“Kagisano Number 10 – Student Funding” – CHE (April 2016)