Politics and higher education – a volatile mix?

I can’t help but have a little admiration for Nicolas Sarkozy. Regardless of whether or not you agree with his positions – he at least seems prepared to actually do something. Not without a little resistance, however. There have been plenty of protests at all levels in response to his education reforms but the latest loosely represents a mutiny by the Grandes Ecoles as reported last month in The Telegraph – http://www.telegraph.co.uk/news/worldnews/europe/france/6941075/Nicolas-Sarkozy-faces-revolt-from-elite-French-colleges.html.

In a nutshell, the Grandes Ecoles are resisting an attempt to force them to take 30 percent to their intake from under-privileged backgrounds. On the one hand, the populist view is that such students are disadvantaged when faced with the extremely challenging entrance exams, on the other that standards will drop if entry requirements are relaxed.

Both views seem valid, but the key battleground may not be at university admissions age but earlier – with a view to driving standards, and aspirations, amongst more diverse students sooner. Or alternatively to focus on diverse entrants to the often expensive preparatory classes rather than the Ecoles themselves which appeared to be Sarkozy’s view just 14 months ago: http://www.javno.com/en-world/sarkozy-tackles-discrimination-in-french-education_215815

Simplicity is a valuable asset

Rankings of anything seem very good at attracting attention, and the simpler they are the more easily and effectively they draw attention. If anyone has ever told a clever joke and then been called upon to explain it you will understand what I am referring to, by the time your audience has understood the joke it has ceased to fulfil its primary purpose – to make people laugh.

There is a great deal of chatter online at the moment – speculation about what newly released rankings might look like, what will be included and what won’t the new THE/Thomson exercise and the CHERPA project through the European Commission are generating particular speculation. The premise on which both of these projects are being discussed is that existing rankings do not fairly measure every aspect of university quality, nor do they recognise the differing nature and structure of different institutions.

Any ranking operated on a global level will be constrained by the quality and quantity of data available and the opinion of its designers and contributors. The worrying trend at the moment is that two underlying assumptions seem to be beginning to resonate throughout this discussion:

1. There is a “perfect solution” – or at least one that will meet with dramatically higher acceptance than those already put forward, and;
2. The stakeholders in rankings are like lemmings and will automatically accept the conclusions of one, or the average of all rankings they consider respectable

The CHE is at the opposite end of the scale to Shanghai and QS methodologies – it gathers masses of data from Germany and surrounding countries but doesn’t actually rank institutions or aggregate indicators – their argument, and perhaps it is a valid one, is that it is not for them to decide what represents quality in the mind of the average stakeholder – particularly students. Fair enough but, broadly speaking, the more proscriptive rankings are not making this assertion either. To my knowledge neither Shanghai Jiao Tong nor QS have ever asserted that their results should be used as the only input to important decisions – the responsibility for such decisions remain the responsibility of the individual making them.

The focus of new developments seems to be on working to the needs and demands of the institutions being evaluated, rather than addressing the needs of the people using them. If such a thing is possible, developing a completely fair and even-handed evaluation, only comparing like against like, is going to become exponentially complex, involving tens, perhaps even hundreds of distinct indicators each engineered in deep technical ways to counter for discipline bias, cultural variety, financial environment, response rate, institution typology, focus, age to a degree that, however transparent the approach is intended to be – its complexity will serve to cloud understanding and the time involved to retrieve and understand results may be off putting.

So the assumptions above seem flawed – it would be irresponsible of any ranking or evaluation to suggest that it is sufficiently complete to be the sole source of data for effective decision making – this increased complexity will promote this illusion rather than allay it and, frankly, the vast majority of people referring to existing rankings are shrewd enough to not take them as more than a single input to their decision making process.

I am, personally, looking forward to seeing what emerges from some of these new projects but in order to achieve some of their bold stated objectives, they are likely to have to sacrifice simplicity, which is not necessarily in the interests of the user.

Rankings coverage

It seems the next 12-18 months are going to be a busy time in the world of rankings and there has been some high profile coverage lately:

Rival rankers rush to market – http://www.theaustralian.com.au/higher-education/rival-rankers-rush-to-market/story-e6frgcjx-1225826091481

Rankled by rankings – http://chronicle.com/article/Rankled-by-Rankings/63786/

You think we’re rankings obsessed? - http://www.insidehighered.com/news/2010/02/01/rankings

The last article references the QS SAFE evaluation of national systems. Interestingly, the Cybermetrics team have adopted the QS SAFE methodology to calculate national standings drawing on data from their own rankings – http://www.webometrics.info/Distribution_by_Country.asp

All is not quiet…

Dear readers,

I must apologize for the silence for the last few months.

Most frequent readers of this blog will know that, the publishing arrangements for our rankings will be changing. In October, THE notified us and, on the same day, the world that they were no longer going to publish our rankings and would be doing something different.

We felt it best to let the initial news sink in before putting forward our position.

QS owns the intellectual property to the previous methodology and all previous data relating to the rankings that have been published in THE for the past six years. The QS World University Rankings will continue to be published in 2010, albeit through a number of new channels which we are working on. At present, there are no plans to alter the methodology, in fact it seems important to maintain some comparability in a time when a number of new and different interpretations are going to emerge. So in 2010, we are focused on improving our engagement with institutions, redesigning some of our data collection systems to be more user-friendly and intuitive, and our work in specific regional and discipline oriented contexts.

It has been extremely busy of late, and keeping the blog as up to date has been a clear challenge. I would welcome any contributions but we will try to keep things going a little more consistently in 2010.

QS Classifications

The THE – QS World University Rankings attract a great deal of interest and scrutiny each year, one piece of frequent feedback is the comparing “apples with oranges” observation. The simple fact is that the London School of Economics bears little resemblance to Harvard University in terms of funding, scale, location, mission, output or virtually any other aspect one may be called upon to consider – so how is it valid to include them both in the same ranking. They do, however, both aim to teach students and produce research and it has always been the assertion of QS and Times Higher Education that this ought to provide a sufficient basis for comparison.

In essence, it is a little like comparing sportspeople from different disciplines in a “World’s greatest sportsperson” or “World’s greatest Olympian” ranking which so frequently emerge. How is it possible to compare a swimmer with a rower with a boxer with a football player? Yet such comparisons have fuelled passionate conversation all over the world. The difference, perhaps, is that in that context those talking are aware of who represents what sport. That is where the classifications come in – they are a component appearing in the tables from 2009 that help the user distinguish the boxers from footballers, so to speak.

The Berlin Principles (a set of recommendations for the delivery of university rankings) assert that any comparative exercise ought to take into account the different typologies of its subject institutions, whilst an aggregate list will continue to be produced it will now feature labels so that institutions (and their stakeholders) of different types can easily understand their performance not only overall but also with respect to institutions of a similar nature.

Based very loosely on the Carnegie Classification of Institutions of Higher Education in the US, but operated on a much simpler basis, these classifications take into account three key aspects of each university to assign their label.

1. Size – based on the (full time equivalent) size of the degree-seeking student body. Where an FTE number is not provided or available, one will be estimated based on common characteristics of other institutions in the country or region in question
2. Subject Range – four categories based on the institution’s provision of programs in the five broad faculty areas used in the university rankings. Due to radically different publication habits and patterns in medicine, an additional category is added based on whether the subject institution has a medical school
3. Research Activity Level – four levels of research activity evaluated based on the number of documents retrievable from Scopus in the five year period preceding the application of the classification. The thresholds required to reach the different levels are different dependent on the institutions pre-classification on aspects 1 and 2.

This will result in each subject institution being grouped under a simple alpha-numeric classification code (i.e. A1 or H3. Table 1 lays out the thresholds for the application of the classifications.

The intention is not to infer a hierarchy – the ranking exists for that purpose – A1 is not a fundamentally preferable classification to G3, but to qualify the subject institutions by broad type with a view to making ranking results more contextually relevant to their increasingly broad audience.

Table 1: Thresholds for application of QS Classifications

			Large	Medium-sized	Small
			>=12,000 FTE Students	>=5,000 <12,000 FTE Students	<5,000 FTE Students
Fully Comprehensive Operational in all 5 faculty areas[2], has a medical school		Research Activity Level[2]	A	E	I
	1	Very High Research Activity	10,000	5,000	2,500
	2	High Research Activity	3,000	1,500	750
	3	Moderate Research Activity	500	250	100
	4	Limited or No Research Activity	0	0	0
Comprehensive Operational in all 5 faculty areas[1]		Research Activity Level[2]	B	F	J
	1	Very High Research Activity	5,000	2,500	1,250
	2	High Research Activity	1,500	750	400
	3	Moderate Research Activity	250	100	50
	4	Limited or No Research Activity	0	0	0
Focused Operational in 3 or 4 faculty areas[1]		Research Activity Level[2]	C	G	K
	1	Very High Research Activity	2,500	1,250	650
	2	High Research Activity	750	400	200
	3	Moderate Research Activity	100	50	50
	4	Limited or No Research Activity	0	0	0
Specialist Operational in 1 or 2 faculty areas[1]		Research Activity Level[2]	D	H	L
	1	Very High Research Activity	2 x mean for specialist areas	2 x mean for specialist areas	2 x mean for specialist areas
	2	High Research Activity	1 x mean for specialist areas	1 x mean for specialist areas	1 x mean for specialist areas
	3	Moderate Research Activity	0.5 x mean for specialist areas	0.5 x mean for specialist areas	0.5 x mean for specialist areas
	4	Limited or No Research Activity	0	0	0

[1] Faculty areas are the 5 faculty areas covered by the THE – QS World University Rankings Academic Peer Review: Arts & Humanities; Engineering & Technology; Life Sciences & Medicine; Natural & Physical Sciences; Social Sciences

[2] Research activity levels are defined against thresholds in terms of number of papers identified in Scopus for a 5 year period

Examples
A1 = Large; Fully Comprehensive; Very High Research Activity (e.g. Harvard, Cambridge, NUS)
A2 = Large; Fully Comprehensive; High Research Activity (e.g. Auckland, University College Dublin)
G1 = Medium-sized; Focused; Very High Research Activity (e.g. Tokyo Institute of Technology)
H1 = Medium-sized; Specialist; Very High Research Activity (e.g. London School of Economics)

2009 THE – QS World University Rankings Complete

Apologies for being silent for so long. Not only have we been exceptionally busy compiling the latest version of the World University Rankings, but I am also pleased to announce that I have become a father for first time – further disrupting my plans to update frequently.

We have finished our final checking and analysis for the 2009 rankings and submitted the needful data to Times Higher Education for publication on 8th October – the Top 200 list will emerge on www.topuniversities.com on the 8th of October with the complete tables to follow on the 9th. What’s more, if all goes to plan, this year’s tables will be interactive, enabling users to add and remove columns, sort by different factors and compare institutions. Busy busy.

This year’s results will be the most stable yet, with the average change in position amongst the top 100 down to 7.4 places from last year’s 11.6 and across the top 500 an average shift of 25 places down from 31. Good news in general terms, then, but there are still some surprises, some interesting new entries, some regional shifts in influence and even changes in the top 10.

Technical challenges with tracking publications and citations for certain institutions.

Tracking all the papers and citations data we need from the Scopus database to fuel our evaluations is quite a challenge and our process has always resulted in some discrepancies between the results we are using and the results that you can actually retrieve from Scopus at given moment. Scopus is an ever-changing database, not only are Elsevier working very hard to add more journals, in more languages and backfilling, but they are alos workign hard to concolidate affiliations and make it easier to retrieve all the data for a given author or institution. The database is vast, however, and the variants are many – apparently MIT, for example at point in time has 1,741 name variants. Additionally, as time goes by, more papers get published and more citations get filed.

Our analysis is based on “custom data” exported from Scopus at a fixed point in time, defined within fixed limits. We use the last five complete years for both papers and citations – that is to say we take a count of all papers published in the five years leading up to December 31st of the previous year and the total of any citations received during the same period. By the time the Times Higher Education – QS World University Rankings are published in October there will 10 more months of papers and publications appearing in the online version Scopus.

The custom data for the forthcoming 2009 analysis amounts to 18Gb of raw XML data – along with this Elsevier provide an affiliation table. This table is an improving lens that we can use to identify the mappings required to retrieve the aggregate data we need. We search this affiliation table for strings that match the universities (or their alternate names) in our database which returns a list of 8 digit affiliate id numbers which we can then use to retrieve and aggregate data from the main data set. If key names are missing from the affiliation table it is very difficult to identify and content that may exist in the main dataset.

Since the publication of the QS.com Asian University Rankings a couple of institutions have come forward and expressed that to some degree or another, data is missing for their institution. This has been discovered thanks to our practice of sharing a “fact file” with institutions prior to publication. Each of them are now working with QS to ensure that any shortfall is rectified in the future.

In future we will be splitting our fact file distribution into two with one comeing out long in advance of publication and then a media briefing which will include the ranking results two days prior to the publication date.

QS.com Asian University Rankings: Beyond the obvious…

I have just returned from a trip to South Korea and Japan where I was presenting the methodology and results of the QS.com Asian University Rankings (AUR) and speaking to a number of universities about the implications of the results in both general and specific terms. Inevitably, as with any ranking upon publication, some institutions are pleased and some are disappointed. The University of Hong Kong at number one seem to be very pleased and their vice-chancellor has been very hospitable and forthcoming, the University of Tokyo… not so much.

These results are not necessarily an omen for the next THE-QS World University Rankings (WUR), however, where Tokyo is reasonably likely once again to assert itself as number one amongst Asian institutions in the global context. There seems a little confusion about that… how can two evaluations from the same organisation yield different results? Well, it’s all about the context. Firstly the methodology for AUR is different from that of WUR, in the narrower context we have been able to gather more data – most notably in the area of exchange programs and we have altered the way we look at publications and citations with a view to being more generous to institutions not operating principally in English. Even without the methodological operations however, the results would not have been the same as WUR because the normalisation of each data point involves the mean and standard deviation of a wildly different dataset.

“Flagship” thinking seems to dominate the thinking of most people in the region – they look at the performance of the top institution in their country, and if it has dropped they assume that either the higher education system in their country is failing and requires reform, or more commonly that the ranking has no value and need not be considered any further.

Digging a little deeper into the performance of Japanese universities in AUR, however, reveals a healthy counterpoint to any observations about the performance of the University of Tokyo – Japanese universities as a cohort have done better in AUR than WUR with 10 top 20 institutions as opposed to only 6 in an Asian extraction from WUR – the implication is that the AUR methodology is indeed, as intended, more accepting of institutions with limited output and teaching in English than WUR. The trend for Japanese universities continues all the way through the results.

Number of Japanese institutions featured within various strata of the AUR and WUR

As the chart shows, performance of Japanese institutions collectively is considerably stronger in AUR than WUR. The same is true of institutions in South Korea and Hong Kong where institutions in Singapore, Mainland China and Taiwan fare less well.

At our session in Tokyo, I was asked… “Have you thought about the customer, won’t it be confusing for potential students?”. We have thought about the users of our website and I would rather have them confused than blindly follow our results anytime. I recently met a student at the NAFSA conference in Los Angeles late last month who sat in on one of our presentations – he said “I have used your rankings to choose a university three times, first Macquarie, now Berkeley and next a university in the Netherlands, but now I am concerned I may have made the wrong choice”. I reassured him that if I ever met anyone who made a crucial life-changing decision based only on the results of a ranking (any ranking) then I would see what I could do to have them committed immediately. The point of rankings is to help inform decisions not to eliminate them.

I found myself on occasion in Seoul drawing on a football (soccer) analogy to explain the differences between AUR and WUR. It is a little bit like the difference between the English Premiership and the UEFA Champions League – the two tournaments require different attributes and different strengths to win, as a result it is not always the team that wins domestically that goes the furthest in the regional tournament. There is no question that both achievements indicate excellent teams and there is not necessarily any way to choose which of the two is actually the better team in the round – just that in the context of the two different competitions one or the other was better suited at the given point in time.

Business schools to blame for the credit crunch?

When travelling around the world on various airlines it is often easy to lose touch with what might be going on at home, so seldom does a UK newspaper pass by your eyes. Yesterday I managed to pick up a copy of the Financial Times (Europe edition no less) at the airport in Seoul before flying to Tokyo – Monday’s edition of course, but nonetheless…

Right under the masthead was the headline “Has the credit crunch tarnished the MBA?”. Turning to page 10 revealed a very interesting article from Della Bradshaw including comment from Deans of a number of leading business schools on the role of business schools in the development of the current financial crisis. I found an online version of the article here – http://www.ft.com/cms/s/0/a2353870-53c4-11de-be08-00144feabdc0.html

Of particular interest is the contrast between comments from the Dean of the Krannert School at Purdue and those from the Dean of IE Business School – who seem to occupy opposite corners in the debate. Despite the opposing views both standpoints seem to be valid. Perhaps business schools have had a part to play in the education of many of the architects of this financial downfall, but they cannot take full responsibility for the MORAL education of their graduates. As the NRA consistently preach, owning a gun does not a murderer make. Business Schools may have had a part in the provision of the tools, but less so in the manner of their use.

The conclusion of the article and those accompanying it seem to be clear though, the MBA needs to evolve, as business is evolving and the general feeling is that this evolution is unlikely to begin at the top US Business Schools.

AIR: A transferable model?

Over the past two days I have had the pleasure of attending the 49th Annual Forum of the Association for Institutional Research in Atlanta. Normally at many of the conferences we attend there is a filtering process to find the right people to talk to, and then we still attract a fairly diverse audience from all levels of a university – the presentations we deliver have to meet these diverse needs. They frequently go well but are far less focused than this.

The very idea that there is a dedicated office in a university collecting, compiling, submitting and analyzing data on themselves, educating the strategy of the institution seems to be a predominately US inspired idea – it happens inrelatively isolated cases elsewhere – but that it has been running an annual conference for 49 years implies extraordinary foresight. Our experience has been of many institutions establishing these offices in response to the requirement to collect and submit data rather than for proprietary business intelligence needs.

The existence of this kind of an office within an institution provides a highly effective, expert conduit for our communication with universities, as has been the case with universities in many countries, but the practice taking broad enough hold to justify a national, regional or even global organisation and conference to support them provides the institutions with a great deal more leverage over third-party evaluating organisations such as ours. Indeed, Robert Morse, of the US News who kindly introduced us to this group has been coming to this event for 10 years – using it as a forum to collect qualified feedback and present and discuss proposed methodological changes. It will be an annual appointment on our calendar from this point forward.

If you work in an office of this nature, in a country where access to such a forum is not yet present, I would strongly recommend you investigate its foundation. Or join an international group.

Check out the US version on www.airweb.org or their international affiliates here www.airweb.org/page.asp?page=577