Category Archives: Education

Below is a quick summary of what I was working on at the OECD in Paris last year. The full paper is now available online here:

• Spaull, N. (2017) Who makes it into PISA? Understanding the impact of sample eligibility using Turkey as a case study (PISA 2003- PISA 2012). OECD Education Working Papers, No. 154, OECD Publishing, Paris. DOI: http://dx.doi.org/10.1787/41d175fc-en

Of the OECD countries that participate in the Programme for International Student Assessment (PISA), Turkey has one of the lowest levels of performance and the highest rates of improvement in PISA scores. However, most analyses have traditionally ignored one vital question: what percentage of 15 year olds in Turkey are eligible for the PISA sample in each wave of PISA? A new OECD Working Paper focuses on this specific question and sheds new light on the performance of Turkey between 2003 and 2012. It shows that the percentage of students that were eligible for PISA in Turkey between 2003 and 2012 nearly doubled from 36% to 68% (using OECD indicators) or from 45% to 80% (using household survey data). This is summarised in Figure 1 below which provides information from the Turkish Demographic and Health Survey (DHS) of 2003, 2009 and 2012.

Figure 1: The educational status and PISA-eligibility of 15-16 year olds in Turkey in DHS 2003, DHS 2008 and DHS 2013

While PISA aims to assess a nationally representative sample of 15 year olds, because PISA is a school-based survey, in reality it excludes all students that are no longer in school (due to drop out or non-enrolment). PISA also only samples 15 year olds if they are in Grade 7 or higher. So PISA is really a nationally representative sample of 15 year old students that are still enrolled in school and are currently in Grade 7 or higher. This might sound like a trivial technicality, and in most wealthy OECD countries like Germany or Japan it is. But some OECD countries (like Turkey and Mexico) and many partner countries (like Vietnam and Indonesia) have high levels of student dropout and delay leading to low levels of PISA sample coverage. As an aside, the new PISA-for-Development initiative aims to also survey out-of-school 15 year olds in some of the developing countries participating in that programme.

Since the beginning of PISA, the OECD has reported the percentage of 15 year olds that are actually eligible for PISA, what is called “Coverage Index 3*.” This statistic is calculated using census and enrolment data in each country and is provided in the overall PISA Reports and Technical Reports for all participating countries. For example, in Turkey in PISA 2003, only 36% of 15 year olds were eligible for PISA. That is to say that PISA 2003 in Turkey is only representative of 36% of the country’s 15 year olds. By comparison, the figure in Germany in 2003 was 93%. Table 1 below provides the Coverage Index 3 rates for a selected group of PISA countries with low levels of sample coverage (Germany and Canada are included as reference countries). From this we can see that a number of partner countries have very low levels of sample coverage, including Costa Rica, Indonesia, Peru and Vietnam, but also that some OECD countries (such as Brazil, Mexico and Turkey) have low levels of sample coverage.

Table 1: The percentage of the total 15 year old population covered by the PISA sampling frame (Coverage Index 3) in selected countries

Given these low levels of sample coverage in some countries, it is reasonable to ask: how would the results change if we included PISA-ineligible 15-year-olds in our calculations? This is the focus of a new working paper published this week, titled “Underestimating Progress and Inequality in Turkey (2003-2012): Using PISA and DHS to form a complete picture of access and quality”. The method and results are briefly summarised below.

The paper applies a new method developed by Spaull & Taylor (2015) which aims to combine statistics on the learning outcomes of 15 year olds that are still in school (using PISA) with data on the number and type of 15 year olds that are not in school (using household-survey data). By assuming that PISA-ineligible students would not have reached PISA Level 2 in Reading and Mathematics – a relatively conservative assumption – we can calculate the percentage of the total population of 15 year olds that reach Level 2 in PISA, rather than only the percentage of those that are still in school. Figure 2 and Figure 3 below provide these breakdowns for Turkey in PISA 2003 and PISA 2012 respectively, and also by gender and socioeconomic subgroups. (Note: ‘Poor40’ means the poorest 40% of 15-16 year olds, and ‘Poor40F’ means poorest 40% of 15-16 year olds that are also female).

Figure 2 Access to Literacy (Level 2) in Turkey 2003 (PISA 2003 and DHS 2003)

Figure 3 Access to Literacy (Level 2) in Turkey 2012 (PISA 2012 and DHS 2013)

Note the results above will be different from those found in PISA reports because these results include PISA ineligible 15-16 year olds in the calculations.

The 7 main findings from the above are as follows:

1. There have been significant increases in PISA sample eligibility over time from around 45% of 15-16 year olds in 2003 to 80% in 2013.
2. If we account for the growth in PISA eligible 15-16 year olds, the improvement in the percentage of 15-16 year olds acquiring Level 2 is between 2 times larger (for Mathematics) and 5 times larger (in Reading) than previously thought.
3. Severe delays in grade progression in Turkey have been virtually eliminated and dropout has declined by 42% over this period.
4. In 2003 the richest group were twice as likely to be eligible for the PISA sample than the poorest group.
5. Although both boys and girls benefitted over the period, girls benefitted the most.
6. The gap in access-to-literacy and access-to-numeracy between rich and poor has not changed and is larger than previously thought.
7. 15-16 year olds in the East region of Turkey are less likely to be eligible for the PISA sample than 15-16 year olds in other regions.

Whether one chooses to use the Coverage Index 3 figures reported by the OECD itself, or those calculated from various DHS surveys, the conclusion is the same: there are large and changing proportions of Turkish students that do not make it into the PISA sampling frame and this has a substantial effect on the validity of inter-country and inter-temporal comparisons. This analysis shows that the gains in Turkey between 2003 and 2012 have actually been even more impressive than if one only looked at PISA data. This method could also usefully be applied to other middle-income and developing countries with high and changing percentages of PISA-eligible 15 and 16 year olds.

References

OECD. (2016). PISA for Development: Benefits for participating countries. PISA for Development Brief (Vol. 2). Paris.

Spaull, N., and Taylor, S., (2015). Access to what? Creating a composite measure of educational quantity and educational quality for 11 African countries. Comparative Education Review. Vol. 58, No. 1.

*There is an error in the Coverage Index 3 values provided in the PISA 2003 Report. The correct Coverage Index 3 values for 2003 can be found in the PISA Technical Report for 2003. See the Working Paper for a full discussion.

Yesterday RESEP had our first internal education workshop of the year and it was a great success. One of the presentations was by Martin Gustafsson on various trends and shocks in the  education system. I found it to be especially important and interesting and asked if I could include the presentation in blog format, which Martin kindly agreed to (see below). For those that don’t know Martin he is easily one of the top 3 most knowledgeable people about the SA education system.

The 2 main points that Martin makes are:

1. There is irrefutable evidence that there has been real progress in learning outcomes in SA over the last decade,
2. That there is a very strange demographic trend in SA (confirmed by 2 independent data sources) that there was a large increase in births in 2004 and 2005 (in the order of 10%) which is extremely peculiar. Note this has significant implications for resource allocation, class sizes etc. This was discussed in the DBE 2016 Sector Review (see excerpt below), but hasn’t been given nearly enough attention.

Regarding #2, from discussions with Martin, who has been involved in reviewing the 2013 SACMEQ results, it seems clear that the 2007 to 2013 SACMEQ trend has indeed been positive for South Africa, and about as large as one could expect (and roughly in line with what has been seen in TIMSS). But these positive trends are smaller than the extremely large improvements, which I’ve argued are implausible, seen in earlier preliminary SACMEQ results presented to the Parliamentary Portfolio Committee on Basic Education in 2016. What the sector urgently needs is the finalising of the 2013 SACMEQ process and the release of official learner results for all SACMEQ countries.

The trends seen in #3 below are from this 2016 paper of Martin’s.

“At the lower primary level, what has clearly occurred is the entrance of a ‘wave’ of larger birth cohorts. In 2011, Grade 1 enrolments increased by 5.2% relative to 2010. In 2012, Grade 2 enrolments increased by 7.0%. By 2015, the increase had reached Grade 5, with enrolments in this grade increasing by 5.2% relative to 2014. As seen in Table 2 in the appendix, substantial increases were seen across all provinces, with the exception of Eastern Cape, which in fact saw a decline . The increases were strongest in Gauteng and Western Cape, almost certainly because these two provinces experienced a combination of two factors, both the demographic change in terms of larger birth cohorts, plus migration into these provinces. By 2015, grades 1 to 7 enrolments had reached 7,1 million in public ordinary schools, the highest figure since 2007 (when the enrolment total was also around 7.1 million). Moreover, total enrolment in just grades 1 to 5 was higher than it had been in any year since 2002. Projections indicate that the increases will continue to be felt in grades beyond Grade 5 in 2016 and beyond. ” Page 9 of this 2016 DBE Sector Report.

*Note in the graph s above (top left) the steady rise in births from 1982 to 1998 are simply because of better data-capturing rather than increased births. It stabilises from about 2000.

For those that haven’t read the DBE 2016 Sector Review I would strongly recommend doing so!

I have been meaning to blog about some new research on access to higher education that was published earlier this week: “Higher Education Access and Outcomes for the 2008 Matric Cohort” (Van Broekhuizen, Van der Berg & Hofmeyr, 2016). I will only highlight some of the key points from the 122-page Working Paper which is really worth reading in its entirety. Essentially the researchers used the matric data from 2008 and followed these students (using their ID numbers) into the higher education system using data from the Higher Education Management Information System (HEMIS). Perhaps the most striking feature is that of the 100 students that started school, only 12 ever access university (9 immediately after matric and 3 later), 6 get some kind of qualification within 6 years and only 4 get a degree within 6 years.

Secondly, that matrics that attend quintile 5 schools (almost all of which charge fees) are four times as likely to access university than those from the poorest 60% of schools (quintiles 1-3), all of which are no-fee schools. However, it’s encouraging to note that of those quintile 1-3 student that do qualify with a bachelor’s pass, more than 63-68% do actually access university, compared to 70% among quintile 5 students.

Much of the paper points to that fact that unequal access to university is rooted in a highly unequal schooling system where access to high-quality schooling largely depends on a family’s ability to pay school fees. If one looks at the cumulative matric average achievement by race one still finds enormous differentials. While 60% of White matric students achieved 60% or more in matric, only 5% of Black African matrics score at or above 60%. And this is only among the students that actually made it to matric which is only slightly more than half the cohort (see this paper).

The last piece of their research that I want to highlight is that the student intake at different universities is vastly different. If one looks at the matric marks of the typical student entering UCT, Stellenbosch, UP or Wits one can see below that they scored 70% or above on average. This is in stark contrast to those students entering TUT, Fort Hare, Uni-Zulu, Walter Sisulu, UWC etc., all of whom have incoming students whose average matric mark is less than 60%. At the Central University of Technology (CUT, in Free State) the average entrant scored 50% in matric.

At the beginning of last year Professor Servaas van der Berg gave a Brown-Bag Lunch Seminar at Stellenbosch University  on “The Distributional Implications of Student Fees.” I include some notable excerpts and graphs below:

“Education has a number of properties which make the analysis of the demand for it both interesting and complex. … (Education is) …a consumption good and a capital good, i.e., although much of the expenditure is justified in terms of the effects on the individual’s income in the future, many of the activities of educational institutions are primarily justifiable in terms of their immediate consumption benefits. Moreover, education affects individuals’ future incomes.” – (Stiglitz 1974: 349)

Perhaps most striking are Van der Berg’s estimates of who actually makes it to university and where they come from in the income distribution. According to these estimates, there are more students attending university from the richest 10% of the income distribution (Decile 10) than from the poorest 80% of the income distribution (Deciles 1-8 combined).

Last month Nico Cloete (from CHET) gave a lecture at SALDRU (UCT) titled: “University Fees in SA: A Story from Evidence.” I include some relevant slides from his presentation:

Anyone who wants to contribute to the debate about university fees needs to grapple with the realities presented by these three papers/presentations. At the end of the day we need to be able to answer the question of where the money will come from. A Graduate tax? Debt? The Education or Health budgets?

The most reasonable (and probably workable) solution that I have heard is that proposed by Prof Van der Berg who suggests that we should use the existing financial services infrastructure (banks) who could provide government-backed grant-loans (my terminology not SVDB’s) to students that qualify for university. It would be a grant that converts into a loan if a student successfully completes their degree and starts earning a decent income. It would still require a huge amount of government finance to provide the surety to banks for students who come from households that earn less than R500,000 (or some threshold). But, unlike with totally ‘free’ education, the students that do successfully complete their degrees would ‘pay-it-forward’ and contribute to the fund used to finance future students. 

Also, as a side-issue, the Fees Commission needs to get a fast-tracked timetable and told to release at least a preliminary report and recommendations before the end of the year. We cannot wait until June next year. The political hot-potato would have been passed along one too many times from VCs to DHET to Treasury and eventually it will just explode. A stitch in time saves nine.

(If you have any additional research suggestions please send me an email and I’ll include them in this post)

Additional inputs from readers: 

“Between the Devil and Deep Blue Sea? The Financing of Higher Education” 3×3 article by Philippe Burger – Sept 2016 (Thanks Marisa!).

Abstract: “Higher-than-inflation increases in student fees since 2009 often are blamed on declining government subsidies to universities. This is not entirely correct, if one considers real per-student subsidies. Fee increases resulted mainly from cost pressures faced by universities due to growing student numbers and a weakening rand. These pressures will not disappear. Eliminating government wastage is not a durable solution and difficult choices cannot be avoided. So, who should pay for increasing costs, students or government – or which combination of these?”

“Kagisano Number 10 – Student Funding” – CHE (April 2016)

I have now moved to Paris to work at the OECD for my TJA Fellowship on PISA data in developing countries. This is essentially an extension of my work on SACMEQ/PASEC and DHS, trying to combine surveys of achievement and attainment to get a composite measure of education system performance.  While I’m away I won’t be commenting as regularly on SA education issues as I usually do. And I will probably shift my focus to other developing countries which is the topic of my research at the OECD. I have already found a number of fascinating things about some PISA countries which I don’t think are widely known or fully appprecaited. There is more than enough included in my previous blog post (11 policy briefs, two synthesis reports and a 200+-page special issue of the SAJCE), not to mention the Volmink Report, for the media and policy-makers to focus on for the next 2 years, let alone 3 months. Keeping up with the nitty-gritty doesn’t make much sense when the underlying issues are not being addressed or taken seriously.

To be honest I’m quite glad to be taking a break from South African education and working on PISA and learning about PISA-for-Development. Clear outcomes, a competent team, and political will. That’ll be nice 🙂

For the last two years we at RESEP have been working on two major education projects: The “Binding Constraints” project (Presidency/EU) and the “Getting Reading Right” project (Zenex Foundation). We launched the two reports on Tuesday last week (my presentation is here). Included below are the two project synthesis reports, a detailed outline on a prospective course (which needs a funder) on “Teaching Reading (& Writing) in the Foundation Phase” and 11 policy briefs. I’ve also included the “Roadmap for Reading” which provides a detailed outline of the practical steps that the Minister of Education could take if she wanted to prioritise reading in the Foundation Phase.

Projects:

1. Identifying Binding Constraints in Education (V2) [Errata from V1 here]
2. Laying Firm Foundations: Getting Reading Right
3. Teaching Reading (and Writing) in the Foundation Phase: A Concept Note

Policy Briefs:

1. Increasing the supply of teacher graduates
2. Education datasets in South Africa
3. Rethinking pre-grade R
4. Is school based assessment in matric achieving its potential?
5. Improving the calibre of school leadership in South Africa
6. The DBE’s workbooks as a curriculum tool
7. Adding randomised control trials (RCTs) to the education research toolkit
8. What the ANAs tell us about socioeconomic learning gaps in South Africa
9. Learning to read and reading to learn
10. Excessive class sizes in the Foundation Phase
11. Building an evidence base for inclusive education in South Africa: Focusing on learners with disabilities
12. Limited support for the Foundation Phase: A Misallocation of district resources

All of the above are also available on the RESEP website here. The 2015 special issue of the South African Journal of Childhood Education (SAJCE) where most of the research was published is available here (ungated) for those who would like to read the full journal articles.

The presentations from the event are available here:

• “An overview of the PSPPD Binding Constraints Research Program” (Servaas van der Berg)
• “Evidence based approaches to literacy: Unravelling the complexity” (Gail Cambell)
• “An overview of the Zenex ‘Getting Reading Right’ research program” (Nic Spaull)
• “How large is the impact of good provincial administration? Estimating the benefit of ‘moving’ to a more functional province” (Stephen Taylor)
• “Utilising the DBE workbooks as a curriculum tool: a review of the Grade 3 workbooks” (Ursula Hoadley & Jaamia Galant)
• “Understanding the distribution and mobility of principals and teachers in South Africa” (Gabrielle Wills)

(I would especially encourage everyone to read through Servaas’ and Gabi’s presentations, they were exceptional!)

There is obviously a lot to be said about these two projects, some of the new research points to very tangible, actionable steps to improve the education system, decrease inequality in outcomes and arguably create a fairer and more efficient education system. Yet it is not at all clear that any of these suggestions will be followed. It was unfortunate that neither the Director General nor the Minister were able to make the report launch. I am aware that both have very busy schedules and that a lot has been happening in education in the last few months. I hope that Servaas will have an opportunity to present this research to them in the coming months. I have also been underwhelmed by the online coverage of the research.

I deliberately do not want to write about the research now since I am currently a little jaded and frustrated about the education research, funding and policy space (perhaps you can tell!). When there are clear, unambiguous and actionable steps that could be taken to improve the education system and they are not taken, this is frustrating. When funders choose to channel millions (billions?) of rands in fruitless directions toward unevaluated projects, it is frustrating. When clear priorities and needs are ignored by national government and local funders (like developing a high-quality course to teach foundation phase teachers to teach reading!), it is frustrating. And perhaps most frustrating of all is the large number of people in provincial and national government that are unable to do the jobs that they have been appointed to do. While there are a number of dedicated and competent public servants and politicians in education, the way that our system is set up means that they have to rely on people who cannot do what they are being asked to do. Those people need to be trained quickly or performance-managed out of the system. That will take courage, strategic leadership and a clear understanding that the status quo is preventing poor children from quality education. Indeed “Weak institutional functionality” or “Insufficient State Capacity” was one of our four binding constraints. Go figure.