Pouring their derision on anything we did… On @NickOsbaldiston and the public role of academics #auspol

English: Australian National University College of Law, Canberra Australia. (Photo credit: Wikipedia)

If you haven’t read Dr Nick Osbaldiston’s blog post over at AusVotes2013, I recommend you do.  In his article, Dr Osbaldiston responds to a sporadic debate that we are having in Australia — the role of academics in public debates.

The University of Melbourne opened in 1853 amid a debate not entirely unlike the debate we are having today.  Where the broader public wanted a practical university to provide practical training in practical practices, an influential group of conservatives wanted the university to be a civilising force during rapid economic growth of Melbourne.  Thus, the inauguration of the classics department.

A similar debate occurred in the 1940s during the establishment of the Australian National University.  This time, it was the Australian Labor Party who was spearheading the institution, claiming that it had ‘not lost sight of the value of culture in the community‘.  The debate at the time was whether the funds going to establish the University was coming at the expense of other sectors.  A member of the Country Party, Winton Turnbull, had this to say about the establishment of a fund to protect the wool industry:

Because you could be the 1% if you wanna… What does NTEU’s ERA report tell us? #auspol

Ever wondered how newspapers found enough time to read all the reports published each day by think tanks, lobbyists, and unions before writing articles about them?  It turns out — and you will be shocked to discover this — that most of those reports don’t get read.  Instead, the journalists dutifully copy and paste the media release sent out accompanying them…

… some of this might be better in a different post that I’ve got cooking up as a result from Monday night’s MediaWatch.  I think MediaWatch has again unquestioningly helped itself to a bunch of assumptions, and that the most interesting parts of the discussion are buried within those assumptions.  Even so, you would imagine that, if a journalist is sent a report attached to a media release, the journalist would read the report before plagiarising from the media release and adding their byline.

Let’s have a look at the National Tertiary Education Union’s new report regarding the Excellence in Research for Australia scheme.

In 2012, the National Tertiary Education Union (NTEU) undertook an exploratory, multi-method study about the implications of the Excellence in Research Australia (ERA) exercise and measures of research performance for Australian university staff. This study relied upon research conducted between April and September 2012, involving:

• A national survey of 39 senior research administrators about the use of ERA and the use of indicators of research performance at universities around Australia;

• Eight recorded and de-identified focus groups at four institutions, and eleven recorded and non-recorded in-depth, semi-structured interviews with participants from five institutions, the recorded sample totalling 50 participants; and

• An NTEU workshop in Melbourne that focused upon the experiences of 35 Early Career Researchers (ECR) and Academics (ECA). [page 3]

For me, it sort of feels like the method (which they will later call their ‘methodology’) doesn’t quite match the proposed task (studying ‘the implications of the ERA exercise and measures of research performance for Australian university staff’).  What it looks like is that you will be able to see to what extent people understand and agree with the ERA.

Lo and behold, that’s what they discovered:

Amongst participants there was a wide divergence of opinions about what the ERA was meant to do, whether it was fulfilling those purposes, and the extent to which it generated public benefit. In contrast to DIISRTE’s claims that there is a ‘broad acceptance of ERA as a rigorous method’ by tertiary education stakeholders, many participants had fundamental concerns with the ERA methodology. In any case, it is important to highlight that the ERA was not universally considered a poor assessment instrument.

Many academics and researchers expressed deep concerns about whether the ERA process was inclusive of different kinds of research output, and the implications for publishing in journals not included in the 2012 ERA Journal List (such as foreign language journals). Some expressed concerns about whether the ERA process was inclusive of non-traditional outputs, such as creative outputs, and other esteem indicators (such as editing non A* ranked journals). In nearly every focus group, an assumption was made was that the ERA process had a disproportionate journal emphasis, and disadvantaged particular disciplines and kinds of research.

In part, this can be contextualised by the fact many academics and researchers had poor access to information about the ERA. However, barriers to information also led many researchers to conflate the ERA with practices and behaviours at various institutions that have either sought to manipulate the ERA outcomes or drawn upon ERA indicators in the performance management of staff. This includes the formal use of the ERA journal rankings over a year after they were abandoned and in spite of some efforts by the ARC to ensure they are no longer used. In limited instances the ERA and the ARC were blamed for institutional research performance practices in fact unrelated to the ERA. [page 17]

This finding, by the way, completely slipped the author’s mind when it came time to writing the executive summary.

The study found that whatever concerns university staff have had about the robustness and probity of the ERA instrument, a pressing concern is what the ERA will be ultimately used for, as the ERA has been integrated into more intense modes of university managerialism. Furthermore, greater investment through the Sustainable Research Excellence (SRE) program will have important implications for Australian universities as workplaces integral to the creation of new knowledge, as was perceived by participants in this study. The policy settings underpinning the ERA and its use directly impact upon the confidence of university researchers in the ERA, as well as their very capacity to undertake broad and diverse types of research. [page 3]

If you make the effort to parse that paragraph, you’ll discover that the statement is in no way supported by the findings in the rest of the report.  The report equivocates on ‘People being interviewed told us that X’ and ‘X is the case’.  Indeed, the earlier point about there being broad confusion about the ERA shows that you can’t draw any firm conclusions about the ERA as a tool.

This problem leaks throughout the report, and the lack of an analytical approach means that the error snowballs.

For example, the report complains that the ERA process does not require universities to consult with the researcher involved about the classification of their research:

It is notable then that the 2010 and 2012 ERA Submission Guidelines exclude any stated obligation for institutions to engage or inform university researchers, even though it is they who are apportioned to FoR codes in institutional submissions and it is their research that is being assessed. [page 22]

The (correct, by the way) claim is made again later on:

As already stated, there is no formal expectation that staff engagement will occur in relation to submission of research output, apportionment or assignment of FoR codes. [page 27]

In this latter instance, it’s insinuated that this lack of engagement is for dodgy reasons:

As already stated, there is no formal expectation that staff engagement will occur in relation to submission of research output, apportionment or assignment of FoR codes. This may provide some protection for staff against allegations of criminal conduct but this would also mean that the capacity to ensure probity about an institution’s submission is significantly diminished. [ibid]

The problem is that the NTEU starts to add fuel to the phony fire between academics and administrators.

It is apparent that there is in fact a fine line between an administrator providing counsel to ensure researchers maximise the university’s ERA score and researchers being improperly directed to assign inaccurate FoR codes. One Level B researcher would have preferred access to her record of research output to ensure that her school was benefiting from her publication record. [page 28]

The Level B researcher never explains for what purpose she should access the ERA submission.  The report strongly suggests that these poor researchers are hardly done by due to these nasty administrators compiling the submission but never provides any argument as to why.

Let’s go through what’s being discussed:

The ERA is like a massive game of Dungeons & Dragons.  The ARC releases a stack of rules by which all universities must abide.  The end goal is to work out whether or not Australian universities are any good at conducting research.  Research itself has a very specific definition.  Furthermore, the goal is to work out where our research strengths are.  We can — and should — debate what we mean by these measures.  How will we know if Australian research is any good?

It’s not, mind, about whether the universities are any good at teaching.  Or pumping out artworks.  Or training first rate lawyers.  Or putting on orchestral pieces for the local cognoscenti.  None of that counts.  We’re talking about research:

The creation of new knowledge and/or the use of existing knowledge in a new and creative way so as to generate new concepts, methodologies and understandings. This could include synthesis and analysis of previous research to the extent that it leads to new and creative outcomes. [Source]

But that’s the puzzle.  A whole lot of Australian ‘researchers’ aren’t pumping out ‘research’ consistent with that definition.

And they get really upset by that.

Some expressed concerns about whether the ERA process was inclusive of non-traditional outputs, such as creative outputs, and other esteem indicators (such as editing non A* ranked journals). In nearly every focus group, an assumption was made was that the ERA process had a disproportionate journal emphasis, and disadvantaged particular disciplines and kinds of research. [page 17]

So some academics are concerned that a process to assess the quality of Australian research doesn’t take into consideration things which aren’t research.  Crazily enough, the ERA has already got a wide number of concessions to non-research activities (esteem measures).

There are two approaches that the ARC could take.  Every university in the country could submit all of its published works to the ARC and they could employ an army of public servants to sift through the publications, assess their eligibility, code them, and then put together batches for independent assessment.  Or — hold on to your hats, people, this is going to get controversial — they could get universities to manage all that crap instead.

I do see it as a problem that the government is increasingly externalising its processes.  Instead of paying for more public servants, they force universities to hire more administrators, which in turn means that universities can’t hire more researchers, which means the government has to provide more money to universities, &c., &c., &c.

On the other hand, getting universities to manage the process does have its benefits.  They have all the systems and knowledge to put together the submission more efficiently than the government can.

But the moment you externalise the process, you allow a problem to enter the system: universities have an interest in presenting the best possible submission that they can, and so they will polish the brightest bits and try to hide the darker bits.  Thus, you need a Dungeon Master’s Compendium of Rules to make sure there’s a level playing field.

The NTEU report intimates that this is a terrible thing.  How dare universities attempt to provide the best possible submission?  Here were the NTEU’s three complaints:

‘Appointing research stars’ – efforts by universities to lift the overall institution or a school’s ERA submission by employing highly productive ‘research stars’, sometimes on a part-time or adjunct basis.

‘Disappearing research’ – where a research manager or senior administrator would ensure certain research outputs would not be evaluated, by apportioning them to FoR codes that did not met the low volume threshold, and without input from the relevant eligible researchers. Sometimes, this would occur through ‘directed manipulation’ with senior administrators directing or advising managers to not enter research, or researchers not to submit output.

‘Horse-trading’ research outputs which involved discussions between university administrators and decision makers to ‘hive-off’ research, i.e. discussing how certain research outputs would be assigned or apportioned multidisciplinary FoR codes to maximise ERA scores in particular institution-preferred FoR codes. [page 26]

The first one is ridiculous.  How the NTEU could be horrified that universities were employing people who could achieve higher levels of research output, I’m not sure.  Perhaps it’s a union thing that they don’t like people who make members look lazy?

The second and third claims are actually the same point.  But let’s put it a different way.  I write the most amazing book on the intersection of legal theory and Batman.  I code it 180122 (Legal theory) and 200213 (Batman studies).  I’m employed by Miskatonic University and their administrators look at my most amazing book and think: ‘We don’t have anybody publishing in Batmanology except for Mark, and we really need more publications in legal theory.  We’ll assign it to be 100% legal theory.’

Nothing dishonest has happened here, but NTEU is suggesting that it has.  What’s more interesting is that NTEU suggests that the university should be negotiating with me personally about the classification of my work, but the classification only makes sense if I know what everybody else is publishing.  I’m sure NTEU would have a fit if everybody’s publication activity were handed out to everybody regardless of need to know.

This is because the NTEU only pays lip service to the ERA’s purpose as an institution-level metric.  It is trying to argue that individual academics have an interest in the ERA submission, but that’s not true.

To sum up:

1. The NTEU makes conclusions not supported by the evidence.  Interviewing people about their perceptions won’t tell you what impacts the ERA is having on university management (especially when you bury the fact that the people being interviewed had a poor understanding of the ERA process);
2. The NTEU conclusions (if valid) only make sense within the framework of particular assumptions about the purpose of the ERA.  Should the ERA assess a broader range of activities than just research?  If so, why?
3. The NTEU complains that universities are hiring people who are capable of greater levels of research output.  Universities should be hiring people capable of higher levels of research output;
4. The NTEU complains that universities are ‘manipulating’ results but does not provide an argument for why this ‘manipulation’ is in any way a problem.

Frankly, it looks like the NTEU is trying to argue against any sort of metric for universities.  There are suggestions that applications for research grants is providing its own measure of research performance (page 36), but the NTEU stops short of endorsing an ‘invisible hand’ approach because, well, a lot more academics would lose their jobs if you based it just on performance in research applications.

The best piece of advice I received regarding the ERA was from a very senior academic who had passed through several different assessment schemes: treat academia like a real job and do your best.  Neither the ERA nor the ERA journal rankings showed anything that people didn’t already expect.  We already knew which journals were the good ones.  We already knew which law schools, philosophy departments, &c., &.c., were the good ones and which were less so.  We just needed some ‘objective’ way of demonstrating it.  If an academic is doing a great job and publishing as they should be, the institution’s ERA process should take care of the rest.  The whinging tends to come disproportionately from people who are worried about demonstrating their research performance to others.

That’s not to say that the ERA process is perfect.  Classics doesn’t have an FOR code, so we should expect to see classics schools wiped out by the ERA (because it will be hard for them to show how they generate research income).  That’s just one example of many systemic problems.

If only the NTEU had been analytical, its report would have been able to show these extremely problematic systemic problems.  Instead, it went down the easy path of having a sook.

 

You go talk to your friends, talk to my friends… @ConversationEDU: Academic opinion vs opinion of academics

This is a particularly difficult blog post to write as it criticises something which, on the whole, I think is a good thing: The Conversation.

During the height of questions about ‘balanced media’ and ‘expert authority’ surrounding climate change and (in Australia, at least) the imposition of a carbon tax/trading scheme, The Conversation was established, supported by various universities and research institutions, to provide an outlet for expert engagement with the community.

The Conversation is an independent source of analysis, commentary and news from the university and research sector — written by acknowledged experts and delivered directly to the public. Our team of professional editors work with more than 4,000 registered academics and researchers to make this wealth of knowledge and expertise accessible to all.

We aim to be a site you can trust. All published work will carry attribution of the authors’ expertise and, where appropriate, will disclose any potential conflicts of interest, and sources of funding. Where errors or misrepresentations occur, we will correct these promptly. [Source]

In practice, this has been difficult to achieve.

Let’s start with ‘written by acknowledged experts’.

In May 2012, The Conversation ran an article called ‘On Iran, Obama shows he loves Israel a little too much‘.  Middle East conflicts are notoriously difficult to cover in opinion/analysis articles.  Regardless of what you write, somebody in the audience will scream about bias.  Just to declare my conflicts, I’m pro-Arab.

The article was written by Michael Brull.  Brull is prominent in Independent Australian Jewish Voices, which is a loose organisation extremely critical of Israel.

None of that is disclosed in his article, which instead asserts the contrary: that he has no relevant affiliations.

Further, his claim to ‘expertise’?  ‘Studying for [sic] Juris Doctor at University of New South Wales’.

I pointed this out quietly to The Conversation at the time.  It was ignored.

Unfortunately, Brull is the most obviously problematic of the contributors.  What happens when academics have axes to grind?

Jane Caro is an academic, but I’ve found it difficult to find anything she’s written that’s been peer reviewed.  More famous as a celebrity than an academic, Jane Caro has a number of pet causes, particularly atheism (she’s one of the Australian pop-atheist crowd) and opposing government support for private schools.  Neither of these causes have anything to do with her academic position: she’s a lecturer in advertising, marketing, and communication at the University of Western Sydney.

In September 2012, she was listed as an ‘expert’ on education reform and the Gonski Report.  While she’s employed by an academic institution, her private opinion was confused with academic opinion.

What’s the difference?

Consider ANU’s Academic Expertise and Public Debate policy.  It states:

When staff members speak within the broad framework of the expertise which led to their employment, or which they have subsequently developed through research and scholarly activities as ANU staff members, they are entitled to use their ANU affiliation as evidence of their expertise on the issues.

The important part there is ‘speak within the broad framework of [their] expertise’.

Jane Caro would be subject to the University of Western Sydney’s Media Policy, which states:

[The University] encourages academic staff to engage with the media to showcase their research, and to offer expert scholarly commentary on topics within their areas of expertise.

But the Gonski Report and education reform are not part of Caro’s research expertise and this was not flagged in the article.

This problem was repeated in a subsequent article, ‘The MLC “scandal”: who cares, and why?‘

Caro writes:

Surely it is only of real interest to a very small number of people; those parents who are forking out as much as $23,000 per year in fees for example. And, yes, noone knows better than I do that taxpayers money has also been involved in the running of this school

For those who didn’t follow the story, it involved the peculiar arrangement of a very well connected group of people in Victoria’s business scene.  Due to a dispute between the recently sacked principal of Methodist Ladies College in Melbourne and the governing board of MLC, a lot of people were startled to see the disputes of a highly influential and under-scrutinised group of people.  We often complain that the media doesn’t hold those in power to account and, when it finally does what it’s supposed to do, Caro complained because it involved those wealthy private schools that she hates.

Perhaps a tenuous link could be made to her position as a ‘communications’ lecturer, but Caro’s not an acknowledged expert in Victorian business.  She’s not even an expert in secondary education.  Yet here’s The Conversation presenting her as an expert the lay audience can trust.

And there are plenty more examples like it.  It would be dreary reading to list all the ones I’ve picked up, but it’s fairly regular that an academic will write on a political issue in which they have little expertise.  The strike rate increases considerably when they publish articles by former politicians turned academics…

More problematic, perhaps, is when The Conversation systemically denies a right of reply to people its articles besmirch.  Companies and organisations have often been targeted by invective from academics, with no indication that those groups were offered a right of reply.  University administrators are routinely targeted for invective by disgruntled academics.  But the article which, with breathtaking audacity, demonstrates this problem most clearly was Professor John Dewar’s article in August 2012, ‘Vice-Chancellor: La Trobe protestors abused freedom of speech‘.

Like many universities, La Trobe is faced with the need to ensure financial viability in the face of budget uncertainty.  It decided to target an underperforming faculty to make it more profitable.  This resulted in — it must be admitted, frankly ignorant — outrage from the students.

As an academic, Professor Dewar was given a platform in The Conversation to make rather odd accusations about the students, including the damning claim that they ‘roam[ed] freely around the campus’.

The undergraduate students — not being ‘acknowledged experts’ — were not provided an opportunity to respond.  The Conversation was, by its nature, incapable of providing balanced coverage to the issue.

The problem is exacerbated when The Conversation tries to shift from a news outlet by academics to a news outlet about academics.  A conversation about the funding structures of universities, for example, won’t necessarily be well informed by asking a bunch of academics what they think about funding cuts.  Similarly, asking the person who flips the burgers at McDonald’s what they think about principles of corporate responsibility in the fast food industry regarding the obesity crisis.  It might sound like an unfair comparison, but most academics don’t research academia.

Although it wasn’t published on The Conversation (much to their credit), the best example of this divide between ‘working at a university’ and ‘knowing how universities work’ is Donald Meyers, Australian Universities: A Portrait of Decline.

The book was rejected by publishers.  Here’s Dr Nathan Hollier, Director of Monash University Publishing, on why they wouldn’t touch it:

Unfortunately, while I think Myers [sic] makes some very important points, I also think the argument needs some re-thinking and a different mode of expression before it could be seriously considered for publication.

In  my view (again) the main problem with this manuscript is that the points made are generally not backed up by evidence. There are too many questionable assertions, generalisations and distinctions and too few direct examples with documentation and detailed explanation.

In the course of documenting these examples and explaining them at greater length there would also – hopefully – be a different tone emerging. While it is good to condemn things that need condemning (such as falling academic standards and the increasing managerialisation and bureaucratisation of the university), it is also good to be aware of why people – including those with views different from your own – think and act in the way they do. Generally, I would suggest, people in the tertiary sector are operating within a complex system which, like the society of which it is part, faces various pressures from within and without. A more extended consideration of these pressures would enable a reading of what has happened and is happening within the sector, including the negative parts of this, that does not wrest [sic] on the assumption that large groups of people are either stupid or ‘bad’.  [Source]

Over 168 pages, Meyers outlines his heroic battle against the forces of terribleness at work within universities:

I have incurred the wrath of management on numerous occasions – without even trying. One of the most amusing, however, was my only face-to-face interaction with the Vice Chancellor. It started with an order to appear at the Headmaster’s office, without notice or explanation, shortly after the conclusion of a meeting of the Vice Chancellor’s Environmental Advisory Committee, which I had the misfortune to chair for two years. It transpired that an administrator
who sat on the committee had visited him in tears allegedly as a result of being abused at the meeting. [p. 135]

The reality is that most of the people who move into [positions of Deputy Vice Chancellors, Pro Vice Chancellors, Executive and Associate Deans, Senior Policy Advisers] are people with academic degrees who would not be competitive in the private sector and who no longer have the desire or competence to succeed at teaching and/or research. In many cases I suspect they have little more than a pathological need to wield power over those who make them feel intellectually inferior. [p. 151]

If the higher education is to be convulsed yet again by reform, let it commence with lavage of the tertiary sector gastrointestinal tract to  expel the resident parasites – the managerialists, the pseudointellectuals, the hangers on, the snake oil salespeople and the ever expanding coterie of Education poseurs. The administration should then be subject to radical liposuction, lap banding and abdominoplasty. Botox will be administered to those who scowl. [p. 165]

There needs to be recognition that we are not all intellectually equal. It is important to know how we rate against others in a variety of pursuits and, just as importantly, we need to know in general terms where our skills may lie. The idea that everyone should go to university is just as detrimental as the constant focus on inflating self-esteem irrespective of performance. […] I can see the hackles and hear the howls of the Educationalists at the mere mention of such ideas. It will be unfair to the socially and economically disadvantaged. [p. 166]

My favourite one is the ‘I abused a person to the point of tears; I was told off and I find this hilarious.’

But it illustrates the point.  Academics study their field.  They don’t study universities.  When The Conversation presents articles by academics discussing the nature of universities, they are no longer giving their academic opinion, they are merely opining.

Finally, there are the articles which are just plain bad.  More often than not, they’re articles where the editorial team felt the need to get an article up which addressed some hot topic of the moment and couldn’t find an academic to write the article quickly enough.  Unfortunately, both of the good examples of this are written by the same person which makes it seem a bit unfair and petty (The Conversation‘s archive system is rubbish and I only happened to remember the name because it popped up twice).

‘Universities left economists ill-prepared for the GFC: Wolf‘ tells the story of a public lecture given by Martin Wolf, the chief economics writer for the Financial Times.  I don’t know many people in Australia who’d know who he is, nor would they be intimately familiar with his various stances on economic ideology.

The article completely fails to explain who he is or what he’s talking about:

Universities and macro-economists still haven’t found a way to solve the crisis of economics that was triggered by the global financial meltdown says economist Martin Wolf.

Speaking to an audience of economics students at the University of Melbourne, Mr Wolf said the real world debate on economics and the global financial crisis was one between Keynesians, Monetarists and Austrians, none of whom were seriously represented in contemporary economic discourse.

Keynesians?  Monetarists?  Austrians?  Unless you’re an economics buff, these words aren’t going to mean a great deal to you.  One of the comments to the article wondered why academics from Austria were having such a hard time being represented in economic discourse…

Worse:

Mr Wolf said economists didn’t know what to do about the mismatch between the rigorous academic framework of economics and the heuristics of applied policy work.

“I don’t think it’s a simple answer, but I think that branch of economics is in pretty serious crisis, as in the thirties.”

Again, this is linked to the problem of The Conversation trying to be about universities rather than by academics.  One part of Wolf’s argument was that universities weren’t developing well skilled applied economists and that the public debate (flooded with Keynesian, Monetarist, and Austrian economic philosophies) wasn’t being informed by academic economists.

The same author wrote ‘Murdoch, Scott defend governance in media‘ which follows a common trend for editor-written pieces.  This article responds to a hot topic: the Managing Director of the ABC, Mark Scott, had made some interesting remarks at Senate Estimates.  It coincided with Rupert Murdoch making similarly interesting remarks at News Corps’ AGM.

There’s only one man alive who can write rapidly and ferociously about anything Rupert Murdoch says.  His name is Stephen Mayne and he is batshit cray-cray.  Unfortunately for The Conversation, he doesn’t have an academic position anywhere.

In order to get articles up quickly, editors will write up an article and then, quite strangely, add the musings of an academic they’ve telephoned for some quotes.  In this example, they chose Bruce Arnold, a lecturer in intellectual property law at the University of Canberra.  Arnold doesn’t really discuss the claims or statements made by Scott or Murdoch, opting instead to opine broadly on the commercial media industry as a whole.

In short, it’s difficult to see how these articles (and others which fit the same format) indicate that The Conversation is a site we can trust for academic opinion.  Between people writing wildly outside their field of expertise, articles spun to be about the research sector, and articles which are cobbled together under extremely demanding time pressures, it’s hard for the lay person to spot where the academic opinion ends and the opinions of people who happen to be academics begin.

I said at the start that I thought The Conversation was, on balance, a great thing.  The articles on science are wonderful and there’s an enthusiastic fun about a number of the articles where you can see how much the academics love writing about their subjects.  It’s when it strays into contentious, political areas that the problems emerge and its those articles which significantly reduce the quality of The Conversation overall.