Why you can’t trust World Economic Forum education rankings

The following article should be appearing on the RESEP website sometime today.

//

WEF rankings on education unreliable

Each year the World Economic Forum releases its Global Competitiveness Report which aims to “assess the competitiveness landscape” and “provide insight into the drivers of their productivity and prosperity.” They furthermore claim that this report “remains the most comprehensive assessment of national competitiveness worldwide.” Included in the report is an indicator of education quality where South Africa performs extremely poorly (132 of 144).

 Much of the work conducted at RESEP focuses on education in South Africa, the quality of that education and the links between the schooling system and the labour market. Martin Gustafsson, one of the researchers at RESEP, has looked into the WEF rankings on education and discusses four salient features which explain why the WEF rankings on education are especially problematic.

    1. Understandably, the 2012-2013 Global Competitiveness Report of the World Economic Forum (WEF) has caused a stir in South Africa as, despite a relatively good overall competitiveness ranking (52 out of 144 countries), a few indicators related to government service delivery, in particular education, put the country amongst the worst in the world. Specifically, in terms of the ‘quality of primary education’ we are at position 132 out of 144, in terms of the net primary enrolment ratio we are at position 115, and in terms of ‘quality of the educational system’ we are at position 140. The 2012-2013 report does not really present anything new. The figures in the previous year’s report (for 2011-2012) are very similar.

 2.      With regard to the educational quality indicators, it is important to bear in mind that the WEF does not make use of any standardised testing system in producing its report. Instead, it makes use of an expert opinion approach. In the case of South Africa, six respondents, all from the business sector, are asked to rate the quality of education along a seven-point scale from very good to very poor. One would expect the South African respondents to rate the quality of South Africa’s schooling poorly for a number of reasons. One is that in South Africa we have good data on our educational quality relative to other countries. In particular, TIMSS 2003 placed South Africa last, with respect to Grade 8 mathematics and physical science, amongst the 20 developing countries that participated (the other African countries participating were Botswana, Egypt, Ghana and Morocco). However, there are around 150 developing countries in the world, many of which have very poor information on the quality of their education systems. One suspects that experts in these countries would not rate their educational quality too poorly as they simply do not have the required information. In SACMEQ 2007, South Africa came ninth out of 15 countries in Grade 6 mathematics. It is noteworthy that although Lesotho did considerably worse than South Africa in SACMEQ, its WEF ranking in the quality of primary schooling indicator is 120, against 132 for South Africa. This illustrates the problem with subjective data on a matter which is relatively amenable to measurement.

 3.      With regard to the primary enrolment ratio, it is important to note that UNESCO’s enrolment ratios (the data source for the WEF) are widely regarded as problematic and often not amenable to useful international comparisons due to the fact that UNESCO calculates its ratios using official enrolment totals and official population totals, in other words information from very different data sources. In many developing countries there are strange discrepancies between the two sets of data. The problem for South Africa is that this discrepancy works in the reverse direction compared to most other developing countries. In South Africa, total population figures for children are simply too high compared to the enrolment totals. In most developing countries, the problem is that enrolment totals are inflated. South Africa’s enrolment ratios in the UNESCO reports appear to be relatively poor, but this means nothing and has confused a lot of people. Enrolment ratios derived from household surveys are a lot more reliable and these indicate that South Africa’s enrolment ratios, at least at the primary and secondary levels, are good by international standards. There is an abundance of literature that shows this. The WEF report itself points to the strangeness of the enrolment ratios it uses. According to the report, at the primary level our enrolment ratio is ranked position 115, but at the secondary level it is ranked 53. This raises an obvious question: How can enrolments at the secondary level be relatively good when at the primary level they are poor, yet the former depends on the latter.

 4.      Lastly, World Bank reports now identify 201 countries in the world.

 Part of Martin’s PhD research involved developing a method to compare the performance of countries on different (sometimes non-overlapping) international assessments of educational achievement. His 2012 Working Paper “More countries, similar results. A nonlinear programming approach to normalising test scores needed for growth regressions” can be found here.