Who watches the watchers?
In the Spring of 2015 the Chartered Association of Business Schools (CABS) published an Academic Journal Guide with the declared intention “to provide a level playing field. Emerging scholars will have greater clarity as to which journals to aim for, and where the best work in their field tends to be clustered. By the same measure, publication in top journals gives scholars recognised currency on which career progress can be based” (p.5)
Clearly, this is a worthy objective, always provided that one can have confidence in the methodology followed and the results claimed for the “findings”. But, and it is a very big but, even if one is confident of the accuracy and objectivity of the published rankings, one needs to be very careful about accepting them as a “recognised currency” if it becomes non-convertible and the only basis for making judgements on career progression. Disturbingly, there is a great deal of anecdotal and some documented evidence to suggest that in many institutions Journal rankings are being used as the main if not sole criterion for making decisions about recognition and career progression. (See, for example, the San Francisco Declaration on Research Assessment (DORA))
A Blog or Editorial is not the place for an evidence-based analysis of how academic “managers” incorporate Journal rankings into their decision-making behaviour but, on the grounds that such rankings may affect the careers of hundreds of academics who publish in journals like JMM, I would like to offer a short “case study” of why great care needs to be exercised in both interpreting and using rankings.
In the 2015 CABS rankings the Journal of Marketing Management was downgraded from 3 to 2 despite the fact that, together with the British Journal of Management, it was the most frequently cited publication for Unit of Assessment 19 (Business Studies) in the recent REF2014 exercise with 200 nominations. In the view of the Assessors who actually read all the papers the great majority of marketing outputs submitted met the criteria for a three rating with some meriting a higher ranking. In light of this very thorough evaluation of research being published by marketing academics in the UK a decision to ‘downgrade’ JMM would seem to be perverse.
In order to establish how this came about, and as the Founding Editor of JMM, I consulted the original Guide which states that:
“The Guide is distinctive in that, unlike other journal guides, it is not based purely on some weighted average of journal metrics. Rather, the Guide reflects the perceptions of the Editors, informed by the Scientific Committee and by expert peers and scholarly associations with whom they consulted as to the relative standing of journals in each subject area. As a consequence, there is no mechanistic metrics based formula that will capture the published ratings.
On occasion, the ratings of some journals, when based purely on such metrics, do not reflect the views of the relevant academic community. Our purpose therefore was to produce a guide that took into consideration this subjective input.
The subject experts (members of the Scientific Committee representing individual subject areas) were provided with a variety of metrics for each journal (detailed below) and were asked to consult widely within their respective subject area academic communities.” (p.5)
My reading of this statement suggests that the metrics were the starting point for a much more rigourous round of peer review involving wide consultation “with their respective subject area academic communities”. In light of the widespread reservations expressed about metrics this would seem to be a sensible safeguard to adopt.
On learning of the downgrade Westburn Publishers, the owner of JMM, made contact with CABS to register their concern about this and to clarify the methodology followed in coming to the decision to downgrade it. Analysis of the metric given to the ‘subject experts’ suggested that it was inaccurate as it was based on incomplete data, but one of the Co-Editors in Chief of the Guide initially dismissed this on the grounds that Westburn had misunderstood the calculation and that his committee would review the journal again in three years’ time. Small comfort to the hundreds of authors whose work had been ‘downgraded’ as a result of the decision.
In parallel with Westburn’s attempts to confirm the validity and reliability of the metric given to its ‘subject experts’ I decided to undertake my own survey of expert opinion and I wrote to 22 senior marketing educators with distinguished publication records and senior management experience. I asked if they would be willing to offer an opinion of the Journal of Marketing Management as an academic Journal, and provided the CABS grading definitions, as well as some basic information about the journal in terms of submission data, recent content and a link to the journal aims and scope. Of the twenty-two persons I approached eighteen responded. Seventeen of these wrote in support of the quality of content published in JMM with only one dissenting opinion. The names and appointments of my supportive respondents are listed below and clearly indicate the “views of the relevant academic community” as required by CABS.
Taken together with my own view I would argue this represents an overwhelming judgement in favour of reinstatement of JMM’s Category 3 status and a letter with the full 17 statements appended was sent to the Chair of CABS on July 3 2015. He indicated that he had “forwarded [this] to the Editorial Team for response as the Management Board is explicitly not involved in issues relating to the evaluation of journals”. No response to me addressing the specifics of my research was received. When I followed up with the Chair of CABS as to why I had not had a response in over 3 months, the eventual response from CABS was “The nature of academic expert discussions means that there will be many interpretations of quality and no two sets of experts will necessarily agree. This disagreement is a natural part of academic debate.”
At a meeting of the Heads of Department held at the annual Conference of the Academy of Marketing in Limerick in early July 2015 a discussion of the CABS Journal rankings was the major item on the agenda as the Journal of Marketing Management is the official Journal of the Academy. One of the two ‘subject experts’ appointed by CABS was also present and was invited to give an account of the consultative procedure taken by them. (Given that the REF Assessors had read all the papers submitted to the Business Studies UoA it is hard to understand why they were not approached for their views). The subject experts had accepted the metrics given to them and held “long conversations” about them before undertaking “wide consultation” with academics based in Europe, the USA and Australasia. This apparently consisted of sending out a list of the marketing journals ranked by the experts with the question ‘Does this look right to you?’ (One academic included in the CABS consultation indicated to me that they were asked to comment on the list within a very short timeframe, with no specific guidance as to what was expected of them.)
The CABS subject expert declined to give any details of this procedure on the grounds of confidentiality but gave me the impression that it had been a very superficial and rushed exercise by comparison with my own survey which included two former editors of the Journal of Marketing, three Past Presidents of the European Marketing Academy (EMAC), two Past Presidents of the Australian and New Zealand Marketing Academy (ANZMAC), and two Past Presidents of the Academy of Marketing.
Unless and until CABS is prepared to release full details of the qualitative process it followed it is difficult to see how they can ignore the judgement of such distinguished, experienced and authoritative marketing academics who replied to my apparently much more carefully controlled survey.
My own objective in publicising this account is the importance of the Latin epigram ‘Quis custodiet ipsos custodies? (Who watches the watchers?), to which the answer must surely be those whose futures and careers are most closely affected by those sitting in judgement of them. It is ironic that of all the academic disciplines one of the least effective in promoting its cause is marketing. The importance or otherwise of what we do is not a self-evident truth apparent to all and we must be much more diligent in creating awareness of our research and its relevance to better understanding and improvements in marketing practice.
Michael J Baker, Founding Editor Journal of Marketing Management
Letters supporting the quality of the material published in JMM were received from:
- Professor George Avlonitis, Athens University of Economics and Business, Greece: Past President EMAC.
- Professor Rod Brodie, University of Auckland Business School, New Zealand: Founding President ANZMAC.
- Emeritus Professor David Carson, University of Ulster, UK: Past President and Chair of the Academy of Marketing.
- Professor Adamantios Diamantopoulos, University of Vienna, Austria: Chaired Professor of International Marketing.
- Professor Zafer Erdogan, Anadolu University, Turkey: Professor of Marketing, Director of the Graduate School of Social Sciences.
- Professor Mark Gabbott, Macquarie University, Australia: Executive Dean, Past President ANZMAC.
- Emeritus Professor Christian Grönroos, Hanken School of Economics, Finland: A “Legend in Marketing” for his research in service and relationship marketing.
- Emeritus Professor Evert Gummesson, Stockholm Business School, Sweden: International pioneer in service, relationships and qualitative methodology.
- Professor Susan Hart, University of Strathclyde, UK: Executive Dean, Strathclyde Business School; former Editor JMM.
- Professor Gillian Hogg, Heriot Watt University, UK: Deputy Principal and Pro Vice-Chancellor.
- Professor Shelby Hunt, The Jerry S Rawls and PW Horn Professor of Marketing, Texas Tech University, USA: Past Editor, Journal of Marketing.
- Professor Hans Kasper, Maastricht University, The Netherlands: Past President EMAC.
- Emeritus Professor Malcolm McDonald, Cranfield University, UK: best-selling author of marketing textbooks.
- Professor Kristian Möller (active Emeritus), Aalto University, Finland: Past President EMAC.
- Professor Nigel Piercy, Swansea University, UK: Dean, School of Management and Professor of Marketing and Strategy.
- Professor Bodo Schlegelmilch, the Vienna University of Economics and Business, Austria: Dean WU Executive Academy.
- Professor Rajan Varadarajan, Texas A&M University, USA: Past Editor Journal of Marketing. (Prof. Varadarajan provided a letter indicating that given his position as VP for Publications for AMA Journals, it would be inappropriate for him to comment on a ranking of scholarly journals, however, as a marketing educator and researcher he found JMM to be ‘an extremely valuable resource’.
This post is licensed under a Creative Commons Attribution 4.0 International License, unless otherwise stated. Third party materials remain the copyright of the original rightsholder.