Getting published in a top five economics journal is a near-requirement for tenure. But it’s a poor measure of research quality within a system that punishes creativity.
Anyone who talks with young economists entering academia about their career prospects and those of their peers cannot fail to note the importance they place on publication in the so-called Top Five journals of economics, henceforth T5: the American Economic Review (AER), Econometrica (ECMA), the Journal of Political Economy (JPE), the Quarterly Journal of Economics (QJE), and the Review of Economic Studies (ReStud).
Anecdotal evidence suggests that T5 publications have a strong influence on tenure and promotion decisions. However, actual evidence on the influence of the T5 is sparse. How strong is this influence? How do junior faculty perceive this influence? Is the discipline justified in its reliance on the T5 as a filter of quality? What are the consequences of this reliance for the health and advancement of economics? This paper answers these and other questions pertinent to the future and well-being of the discipline.
Documenting the Power of the T5
We find strong evidence for the influence of the T5. Without doubt, publication in the T5 is a powerful determinant of tenure and promotion in academic economics. Analyzing longitudinal data on the employment and publication histories of tenure-track faculty hired by the top 35 U.S. economics department between 1996—2010, we find that T5 publications greatly increase the probability of receiving tenure during the first spell of tenure-track employment and through the seventh year of such employment. We supplement these results with estimates from duration analyses that show that publishing three T5 articles is associated with a 370% increase in the rate of receiving tenure, compared to candidates with similar levels of publications in non-T5 journals. Analogously, we estimate that candidates with one or two T5 publications experience increases in the rate of tenure of 90% and 260%, respectively. The estimated effects of publication in non-T5 journals pale in comparison.
This finding of strong T5 influence is corroborated by results from a survey of current Assistant and Associate Professors hired by the top 50 U.S. economics departments. On average, junior faculty rank T5 publications as having the greatest influence on their tenure and promotion outcomes, outranking seven different areas of research and teaching performance.
While these results provide evidence for the large influence of T5 publications relative to other performance areas, they do not shed light on whether the T5’s influence operates through channels that are independent of article impact and quality. An elicited thought experiment designed to tease-out the confounding influence of inter-journal differences in article quality provides an answer. Survey respondents were asked to report the probability that their department would award tenure or promotion to a hypothetical individual with T5 publications over another hypothetical individual who was identical to the first individual in every respect except that he/she had published the same number and quality of articles in non-T5 journals. If the T5 influence operated solely through differences in article impact and quality, the expected reported probability would be 0.5. Deviations in the mean reported probability from 0.5 in favor of the T5 candidate signal that the T5 influence operates through channels that are independent of article quality.
The results show large and statistically significant deviations in favor of the T5 candidate. On average, respondents from the top 10 departments believe that the T5 candidate would receive tenure with a probability of 0.89. The mean reported probability rises for lower-ranked departments, with its value peaking at 0.93 for departments ranked 31-40.
These results reveal that there exists a widely-held belief among junior faculty at the top 50 departments that the same quantity and quality of articles will yield rewards at vastly different rates depending on whether their articles appear in T5 or non-T5 journals. This expectation is a rational one. Faculty form beliefs based on past decisions, and the empirical evidence on past decisions indicate that decisions are strongly influenced by publishing in the T5.
The T5 as a Filter of Quality
We evaluate the quality of the T5 as a filter of research influence and quality using citations as a proxy for influence. Similar to Hamermesh (2018), we find considerable intra-journal heterogeneity and inter-journal overlap in citations. Moreover, citations accrue differently between the T5 journals. The overlap between T5 and non-T5 journals increases considerably when one compares non-T5 journals to the less-cited T5 journals. As a case in point, while the median Review of Economics and Statistics (ReStat) article ranks in the 38th percentile of the overall T5 citation distribution, its rank improves to the 48th percentile when the comparison set is restricted to articles in JPE, ECMA, and ReStud. The median ReStat article outranks the median-cited article in the combined JPE and ReStud distribution.
Restricting the citation analysis to the top of the citation distribution yields the same conclusion: many non-T5 articles outperform T5 articles, especially T5 articles in the less-cited T5 journals. Among the top 1% most-cited articles in our citations database,[1] 13.6% were published by three non-T5 journals. Each of the three journals produced more top 1% articles than ReStud, and two of the three journals produced at least as many top 1% articles as JPE. ReStud is outranked by six additional non-survey non-T5 journals, which together contributed a further 16% to the pool of top 1% articles.
The current practice of relying on the T5 has weak empirical support if judged by its ability to produce impactful papers as proxied by citation counts. Moreover, academics who impose the T5 standard don’t follow it themselves. They primarily publish in, read, and cite non-T5 journals, as will the candidates who survive the T5 filter and become tenured faculty.
Consequences of the Tyranny
Relying on rankings rather than reading to promote and reward young economists subverts the essential process of assessing and rewarding original research. Using the T5 to screen the next generation of economists incentivizes professional incest and creates clientele effects whereby career-oriented authors appeal to the tastes of editors and biases of journals. It diverts their attention away from basic research toward blatant strategizing about lines of research and favored topics of journal editors with long tenures. It raises entry costs for new ideas and persons outside the orbits of the journals and their editors. An over-emphasis on T5 publications perversely incentivizes scholars to pursue follow-up and replication work at the expense of creative pioneering research since follow-up work is easy to judge, is more likely to result in clean publishable results, and hence is more likely to be published.[2] This behavior is consistent with basic common sense: you get what you incentivize.
Discussion
In light of the many adverse and potentially severe consequences associated with reliance on the T5, we believe it unwise for the discipline to continue using publication in the T5 as a measure of research achievement and as a predictor of future scholarly potential. The practice judges papers by the company they keep, i.e., the general quality of the papers published in a journal and not by the quality of the paper itself. Our findings should spark a serious conversation in the profession about developing implementable alternatives to judge quality research. Such solutions would necessarily need to de-emphasize the role of the T5 in tenure and promotion decisions, and re-distribute the signaling function among more high-quality journals.
However, a proper solution to the tyranny will likely involve more than a simple re-definition of the T5 to include a handful of additional influential journals. A better solution will need to address the flaw that is inherent in the practice of judging a scholar’s potential for innovative work based on a track record of publications in a handful of select journals. The appropriate solution to the problem will require a significant shift from the current publications-based system of deciding tenure, to a system that emphasizes departmental peer-review of a candidate’s work. Such a system would give serious consideration to unpublished working papers and to the quality and integrity of a scholar’s work. By closely reading published and unpublished papers rather than counting placements of publications, departments would signal that they both acknowledge and adequately account for the greater risk associated with scholars working at the frontiers of the discipline.
A more radical proposal is to shift publication away from the current fixed format journals and towards an open source arXiv or PLOS ONE format.[3] Such formats facilitate the dissemination rate of new ideas and provide online real-time peer review for them. Discussion sessions would vet criticisms and provide both authors and their readers with different perspectives. Shorter, more focused papers would stimulate dialogue and break editorial and journal monopolies. Ellison (2011) notes that online publication is already being practiced by prominent scholars. Why not broaden the practice and encourage spirited dialogue and rapid dissemination of unique thought?
In any event, the profession should de-emphasize crass careerism and promote creative activity. Short tenure clocks and reliance on the T5 to certify quality does just the opposite. It centralizes power in the hands of a small group of editors, discourages open discussion and stifles dissent and debate. In the long run, the profession will benefit from application of more creativity-sensitive screening of its next generation.
References
- Eisen, Michael. 2013. “The Past, Present and Future of Scholarly Publishing.” Remarks by Michael Eisen, co-Founder of Public Library of Science (PLOS), at the Commonwealth Club in San Francisco. http://www.michaeleisen.org/blog/?p=1346.
- Ellison, Glenn. 2011. “Is Peer Review in Decline?” Economic Inquiry 635-657.
- Hamermesh, Daniel. 2018. “Citations in Economics: Measurement, Uses, and Impacts.” Journal of Economic Literature 115-156.
- Vale, Ronald D. 2015. “Accelerating Scientific Publication in Biology.” Proceedings of the National Academy of Science 13439-13446.
Footnotes
[1] The database is comprised of citations to all articles published by 25 top economics journals between 2000—2010
[2] See the discussion at https://www.aeaweb.org/webcasts/2017/curse
[3] See Vale (2015) for a discussion of the use of arXiv in physics. See Eisen (2013) for remarks on PLOS ONE by Michael Eisen, its co-founder.
