QS.com Asian University Rankings: Beyond the obvious…

I have just returned from a trip to South Korea and Japan where I was presenting the methodology and results of the QS.com Asian University Rankings (AUR) and speaking to a number of universities about the implications of the results in both general and specific terms. Inevitably, as with any ranking upon publication, some institutions are pleased and some are disappointed. The University of Hong Kong at number one seem to be very pleased and their vice-chancellor has been very hospitable and forthcoming, the University of Tokyo… not so much.

These results are not necessarily an omen for the next THE-QS World University Rankings (WUR), however, where Tokyo is reasonably likely once again to assert itself as number one amongst Asian institutions in the global context. There seems a little confusion about that… how can two evaluations from the same organisation yield different results? Well, it’s all about the context. Firstly the methodology for AUR is different from that of WUR, in the narrower context we have been able to gather more data – most notably in the area of exchange programs and we have altered the way we look at publications and citations with a view to being more generous to institutions not operating principally in English. Even without the methodological operations however, the results would not have been the same as WUR because the normalisation of each data point involves the mean and standard deviation of a wildly different dataset.

“Flagship” thinking seems to dominate the thinking of most people in the region – they look at the performance of the top institution in their country, and if it has dropped they assume that either the higher education system in their country is failing and requires reform, or more commonly that the ranking has no value and need not be considered any further.

Digging a little deeper into the performance of Japanese universities in AUR, however, reveals a healthy counterpoint to any observations about the performance of the University of Tokyo – Japanese universities as a cohort have done better in AUR than WUR with 10 top 20 institutions as opposed to only 6 in an Asian extraction from WUR – the implication is that the AUR methodology is indeed, as intended, more accepting of institutions with limited output and teaching in English than WUR. The trend for Japanese universities continues all the way through the results.

Number of Japanese institutions featured within various strata of the AUR and WUR

As the chart shows, performance of Japanese institutions collectively is considerably stronger in AUR than WUR. The same is true of institutions in South Korea and Hong Kong where institutions in Singapore, Mainland China and Taiwan fare less well.

At our session in Tokyo, I was asked… “Have you thought about the customer, won’t it be confusing for potential students?”. We have thought about the users of our website and I would rather have them confused than blindly follow our results anytime. I recently met a student at the NAFSA conference in Los Angeles late last month who sat in on one of our presentations – he said “I have used your rankings to choose a university three times, first Macquarie, now Berkeley and next a university in the Netherlands, but now I am concerned I may have made the wrong choice”. I reassured him that if I ever met anyone who made a crucial life-changing decision based only on the results of a ranking (any ranking) then I would see what I could do to have them committed immediately. The point of rankings is to help inform decisions not to eliminate them.

I found myself on occasion in Seoul drawing on a football (soccer) analogy to explain the differences between AUR and WUR. It is a little bit like the difference between the English Premiership and the UEFA Champions League – the two tournaments require different attributes and different strengths to win, as a result it is not always the team that wins domestically that goes the furthest in the regional tournament. There is no question that both achievements indicate excellent teams and there is not necessarily any way to choose which of the two is actually the better team in the round – just that in the context of the two different competitions one or the other was better suited at the given point in time.