Prejudice and collusion, not simply research quality, drive journal citations
Economists typically make use of citations to measure the alleged quality of researchers, institutions, journals, or disciplines. Such a perspective aligns well with some standing presumptions on the nature of the scientific process prevalent in economics—for instance the idea that the best research is done in the top departments or that economic research is superior to research in neighboring fields like sociology or political science. However, it systematically neglects the fact that citation patterns are driven by institutional constraints, reputational concerns and disciplinary identity as recent research funded by INET indicates.
Institutional Constraints
In the theoretical part of our project, we show that scarcity of journal space in combination with a competitive, ranking-oriented research culture is not sufficient to ensure that high-quality contributions find their way to the top of the discipline. Quite the contrary, the scarcity of journal space in a competitive environment may quickly lead to perverse crowding-out effects for high-quality contributions even in the absence of factors generally recognized as distorting—like the role of social ties and networks or the importance of the topic under study.
In our model we show that assuming ambitious authors, who only have imperfect knowledge on the relative merit of their contributions, is sufficient for producing such distorting bottlenecks (Kapeller & Steinerberger 2016, see Figure 1). This latter finding resembles a general property of the current academic publishing process and implies that citation-based quality measures and other criterions of visibility should be taken with caution as they might lead us to miss some great ideas.
Figure 1: The distribution of paper quality (orange) and rejection behavior by editors (purple) in an agent-based model of the submission process
Combining these findings with insights on the social structure of the economics discipline supports the indication that the selection of high-quality papers that eventually remain unpublished or ignored in economics is not random: indeed, the strong impact of established hierarchies in the economics discipline (e.g. Hodgson & Rothman 1991, Medoff 2006, Fourcade et al. 2015), social networks (Colussi 2018), political orientation (D’Ippoliti 2017) and theoretical affiliation (Aistleitner et al. 2017) is to be expected. Such an interpretation is therefore completely in line with the results achieved by James Heckman and Sidharth Moktan on the gatekeeping-power of top journals in the discipline of economics (Heckman & Moktan 2018)
In addition, we find that the importance of journal rankings within an academic discipline further strengthens established hierarchies by establishing an additional positive feedback-loop in the process of academic reproduction. We document this by pointing to the immediate rise in submission to journals as a consequence of being included in an important ranking (Aistleitner et al. 2018) as well as by showing how the prestige of journals—relative to individual authors and articles—plays a greater role after the introduction of such rankings (Aistleitner et al. 2017)
Reputational Concerns
To provide a novel perspective on the allegedly strong internal stratification within economics, we inspect the share of reciprocal citations among top journals economics relative to other disciplines. In doing so, we ask to what extent an average paper in a top journal draws on other contributions from established top journals. Our archetypical finding is that the research discourse at the “top” of economics is much more concentrated than in the neighboring social science disciplines, with the average share of reciprocal citations within top journals standing at about one quarter of all citations—far beyond the ratio measured for the other disciplines in our sample (Figure 2).
Figure 2: Citation patterns across four disciplines
While such a high degree of reliance on contributions from top journals could theoretically be solely due to their superior quality, reputational concerns—top journals are cited, because those who cite them want to signal that they also belong to the “top”—provide an alternative explanation for the observed patterns. The latter interpretation is not only much more easily aligned with related findings on the sociology of the economics profession, but it is supported by recent studies on the “gatekeeping” function of top journals: in this account, generalist top journals effectively decide which topics and approaches from more specialized fields of the discipline should gain broader reception.
Disciplinary Identity
The disciplinary culture in economics does not put much emphasis on diversity. While a lack of diversity in terms of demography is well documented in economics (e.g. Bayer & Rouse 2016), Fourcade et al. (2015) recently studied the role of intellectual diversity in economics from a disciplinary angle. By analyzing both attitudes towards interdisciplinary research as well as interdisciplinary citation patterns in economics, sociology, and political science, Fourcade et al. show that economists are comparatively less inclined to import findings from other disciplines and also have less trust towards interdisciplinary approaches and research strategies. These findings are highly consistent with past results from data-driven research on citation patterns (e.g. Figure 3), which document the solitary character of economics in terms of disciplinary location.
Figure 3: The degree of “self-referentiality” across different disciplines
In contrast, the typical approach to citations in economics—to interpret them as a measure of performance—implies a competitive setting, where different disciplines are to be evaluated with regard to their “export performance,” or in the words of a practitioner:
“What is interesting to think about are the terms of trade between economics and all these other disciplines. We are clearly a net exporter to political science and sociology.” (Wolfers 2010, 30)
Such a stance on the subject seems represents a lop-sided approach to the data as it is equally possible that the alleged excellence of economics (measured in terms of net citations exports to other disciplines) is mostly based on its ignorance of other disciplines (as citation-imports from neighboring fields are exceptionally low).
In our paper, we show that the same argument also holds for different paradigms in economics: we find that alternative, heterodox approaches do consider mainstream economic research, while mainstream economics largely neglects contributions from heterodox sources. Here, again mainstream appears to be superior since it is cited by competing approaches, but does itself not rely on external input.
Figure 4: Mutual citations between mainstream and heterodox economics
A more interdisciplinary perspective, that is also incorporating interdisciplinary citation-flows, would lend additional support to the interpretation that the neglect of heterodox ideas is often due to either prejudice or ignorance as heterodox contributions seem to reference an over-proportional share of citations from other disciplines.
What Do Citations Tell Us?
Our research shows how the standard interpretation of citations in economics—as a signifier of the quality of the cited contribution—is fragile. Instead, citation patterns are determined by more than one factor. And, indeed, many of these factors—like social networks, existing hierarchies, paradigmatic identities or reputational concerns—relate to concerns of belonging that can actively distort and dominant the impact of the intrinsic quality of a manuscript. It is at this point that our findings go beyond the narrow realm of economics, but, rather, carry some general implications for established institutional routines in science.
References
Aistleitner, M., Kapeller, J. & S. Steinerberger (2017): Citation Patterns in Economics and Beyond. Forthcoming in: Science in Context.
Aistleitner, M., Kapeller, J. & S. Steinerberger (2018): The Power of Scientometrics and the Development of Economics. Journal of Economic Issues, 52(3): 816-834.
Bayer, A. & C. E. Rouse (2016): Diversity in the Economics Profession: A New Attack on an Old Problem. Journal of Economic Perspectives, 30(4): 221- 242.
Colussi, T. (2018): Social Ties in Academia: a Friend is a Treasure. The Review of Economics and Statistics, 100(1), REST_a_00666–50. http://doi.org/10.1162/rest_a_00666
D’Ippoliti, C. (2018): “Many-Citedness”: Citations Measure More Than Just Scientific Impact. INET Working Paper #57 https://www.ineteconomics.org/uploads/papers/WP_57-DIppoliti-revised.pdf
Fourcade, M., Ollion, E., & Y. Algan. (2015): The superiority of economists. Journal of Economic Perspectives, 29(1): 89-114.
Heckman, J. & Moktan, S. (2018): Publishing and Promotion in Economics: The Tyranny of the Top Five. Institute for New Economic Thinking, Working Paper 82 https://www.ineteconomics.org/research/research-papers/publishing-and-promotion-in-economics-the-tyranny-of-the-top-five ; NBER Working Paper 25093.
Hodgson, G. M. & H. Rothman. (1999): The editors and authors of economics journals: a case of institutional oligopoly? Economic Journal, 109(453): F165-F186.
Kapeller, J. & S. Steinerberger (2016): Emergent Phenomena in Scientific Publishing: A Simulation Exercise. Research Policy, 45(10): 1945-1952.
Medoff, M. H. (2006): Evidence of a Harvard and Chicago Matthew Effect. Journal of Economic Methodology, 13(4): 485–506.
Wolfers, J. (2010): Interview in Region Focus, Second Quarter 2010, Federal Reserve Bank of Richmond, online available at https://www.richmondfed.org/~/media/richmondfedorg/publications/research/re gion_focus/2010/q2/pdf/interview.pdf (accessed 13.03.2017).
