Using single impact metrics to assess research in business and economics: why institutions should use multi-criteria systems for assessing research
Keywords:
Impact factor, Article influence, Business journals, Research assessment, Faculty promotion, TenureAbstract
PurposeDespite the general recommendation of using a combination of multiple criteria for research assessment and faculty promotion decisions, the raise of quantitative indicators is generating an emerging trend in Business Schools to use single journal impact factors (IFs) as key (unique) drivers for those relevant school decisions. This paper aims to investigate the effects of using single Web of Science (WoS)-based journal impact metrics when assessing research from two related disciplines: Business and Economics, and its potential impact for the strategic sustainability of a Business School.
Design/methodology/approachThis study collected impact indicators data for Business and Economics journals from the Clarivate Web of Science database. We concentrated on the IF indicators, the Eigenfactor and the article influence score (AIS). This study examined the correlations between these indicators and then ranked disciplines and journals using these different impact metrics.
FindingsConsistent with previous findings, this study finds positive correlations among these metrics. Then this study ranks the disciplines and journals using each impact metric, finding relevant and substantial differences, depending on the metric used. It is found that using AIS instead of the IF raises the relative ranking of Economics, while Business remains basically with the same rank.
Research limitations/implicationsThis study contributes to the research assessment literature by adding substantial evidence that given the sensitivity of journal rankings to particular indicators, the selection of a single impact metric for assessing research and hiring/promotion and tenure decisions is risky and too simplistic. This research shows that biases may be larger when assessment involves researchers from related disciplines – like Business and Economics – but with different research foundations and traditions.
Practical implicationsConsistent with the literature, given the sensibility of journal rankings to particular indicators, the selection of a single impact metric for assessing research, assigning research funds and hiring/promotion and tenure decisions is risky and simplistic. However, this research shows that risks and biases may be larger when assessment involves researchers from related disciplines – like Business and Economics – but with different research foundations and trajectories. The use of multiple criteria is advised for such purposes.
Originality/valueThis is an applied work using real data from WoS that addresses a practical case of comparing the use of different journal IFs to rank-related disciplines like Business and Economics, with important implications for faculty tenure and promotion committees and for research funds granting institutions and decision-makers.
Downloads
References
Journal Impact Factors (2018), Journal Citation Reports Science and Social Science Edition.
AACSB (2012), “Impact of research: a guide for business schools, electronic document”, available at: https://www.aacsb.edu/insights/publications/impact-of-research-a-guide-for-business-schools.
AACSB (2020), “AACSB 2020 Guiding principles and standards for business accreditation”, Electronic document, available at: https://www.aacsb.edu/educators/accreditation/business-accreditation/aacsb-business-accreditation-standards.
Adams, J., McVeigh, M., Pendlebury, D. and Szomszor, M. (2019), “Profile, not metrics”, Institute for Scientific Information, Clarivate Analytics, available at: https://clarivate.com/webofsciencegroup/campaigns/profiles-not-metrics/.
Adler, N. and Harzing, A. (2009), “When knowledge wins: transcending the sense and nonsense of academic rankings”, Academy of Management Learning and Education, Vol. 22 No. 1, doi: 10.5465/amle.2009.37012181.
Arendt, J. (2010), “Are article influence scores comparable across scientific fields?”, Issues in Science and Technology Librarianship, Vol. 60, Winter, pp. 1-14, doi: 10.5062/F4FQ9TJW.
Azar, O.H. (2009), “The influence of Economics articles on Business Research: analysis of journals and time trends”, The Journal of Industrial Economics, Vol. 57 No. 4, pp. 851-869.
Bergstrom, C.T. (2007), “Eigenfactor: measuring the value and prestige of scholarly journals”, College and Research Libraries News, Vol. 68 No. 5, pp. 314-316.
Bergstrom, C.T. and West, J.D. (2008), “Assessing citations with the Eigenfactor metrics. Neurology”, Vol. 71 No. 23, pp. 1850-1851.
Bergstrom, C.T., West, J.D. and Wiseman, M.A. (2008), “The Eigenfactor metrics”, The Journal of Neuroscience, Vol. 28 No. 45, pp. 11433-11434.
Brito, R. and Rodríguez-Navarro, A. (2019), “Evaluating research and researchers by the journal impact factor: is it better than coin-flipping?”, Journal of Informetrics, Vol. 13 No. 1, pp. 314-324.
Brown, T. and Gutman, S.A. (2019), “Impact factor, eigenfactor, article influence, Scopus SNIP, and SCImago journal rank of occupational therapy journals”, Scandinavian Journal of Occupational Therapy, Vol. 26 No. 7, pp. 475-483.
Cancino, C.A., Merigó, J.M., Torres, J.P. and Díaz, D. (2018), “A bibliometric analysis of venture capital research”, Journal of Economics, Finance and Administrative Science, Vol. 23 No. 45, pp. 182-195.
Carey, R.M. (2016), “Quantifying scientific merit: is it time to transform the impact factor?”, Circulation Research, Vol. 119 No. 12, pp. 1273-1275.
Chang, C.-L., McAleer, M. and Oxley, L. (2011), “How are journal impact, prestige and article influence related: an application to neurosciences”, Journal of Applied Statistics, Vol. 38 No. 11, pp. 2563-2573.
Chang, C.-L., McAleer, M. and Oxley, L. (2013), “Journal impact factor, eigenfactor, journal influence, and article influence”, Tinbergen Institute Discussion Paper TI2013-002.
Chang, C.-L., Massoumi, E. and McAleer, M. (2016), “Robust rankings of journal quality: an application to economics”, Journal of Econometric Reviews, Vol. 35 No. 1, pp. 50-97.
Davis, P.M. (2008), “Eigenfactor: does the principle of repeated improvement result in better estimates than raw citation counts?”, Journal of the American Society for Information Science and Technology, Vol. 59, pp. 2186-2188.
Dorta-González, P. and Dorta-González, M.I. (2013), “Comparing journals from different fields of science and social science through a JCR subject categories normalized impact factor”, Scientometrics, Vol. 95, pp. 645-672.
Eigenfactor (2009), “EigenfactorTM score and article influence TM score: detailed methods”, available at: http://www.eigenfactor.org/methods.pdf.
Elkins, M.R., Maher, C.G., Herbert, R.D., Moseley, A.M. and Sherrington, C. (2010), “Correlation between the journal impact factor and three other journal citation indices”, Scientometrics, Vol. 85, pp. 81-93.
Fersht, A. (2009), “The most influential journals: impact factor and eigenfactor”, Proceedings of the National Academy of Sciences, Vol. 106 No. 17, pp. 6883-6884, doi: 10.1073/pnas.0903307106.
Fischer, C., Malycha, C.P. and Schafmann, E. (2019), “The influence of intrinsic motivation and synergistic extrinsic motivators on creativity and innovation”, Frontiers of Psychology, Vol. 10, Art. 37, pp. 1-15, doi: 10.3389/fpsyg.2019.00137.
Franceschet, M. (2010), “Journal influence factors”, Journal of Informetrics, Vol. 4 No. 3, pp. 239-248.
Garfield, E. (1972), “Citation analysis as a tool in journal evaluation”, Science, Vol. 178 No. 4060, pp. 471-479.
Garfield, E. (2006), “The history and meaning of the journal impact factor”, The Journal of the American Medical Association, Vol. 295 No. 1, pp. 90-93.
Ghoshal, R. (2005), “Bad management theories are destroying good management practices”, Academy of Managament Learning and Education, Vol. 4 No. 1, pp. 75-91.
Gibbons, R. and Kaplan, R.S. (2015), “Formal measures in informal management: can a balanced scorecard change a culture?”, American Economic Review, Vol. 105 No. 5, pp. 447-451.
Gomes, O. and Frade, J. (2019), “Fool me once: deception, morality and self-regeneration in decentralized markets”, Journal of Economics, Finance and Administrative Science, Vol. 24 No. 48, pp. 312-326.
González-Pereira, B., Guerrero-Bote, V.P. and Moya-Anegón, F. (2010), “A new approach to the metric of journals' scientific prestige: the SJR indicator”, Journal of Informetrics, Vol. 4 No. 3, pp. 379-391.
Gorraiz, J., Wieland, M. and Gumpenberger, C. (2017), “To be visible, or not to be, that is the question”, International Journal of Social Sciences and Humanity, Vol. 7 No. 7, pp. 467-471.
Harzing, A.W. (2019), “Two new kids on the block: how do crossref and dimensions compare with Google scholar, microsoft academic, Scopus and the web of science”, Scientometrics, Vol. 120, pp. 341-349.
Harzing, A.W. (2020), “Everything you always wanted to know about research impact”, in Clark, T. and Wright, M. (Eds), How to Get Published in the Best Management Journals, Edward Edgar Publishing, pp. 127-141.
Harzing, A.W. and Van der Wal, R. (2009), “A Google Scholar h-index for journals: an alternative metric to measure journal impact in economics and business”, Journal of the American Society for Information Science and Technology, Vol. 60 No. 1, pp. 41-46.
Haustein, S. and Larivière, V. (2015), “The use of bibliometrics for assessing research: possibilities, limitations and adverse effects”, in Welpe, I., Wollersheim, J., Ringelhan, S. and Osterloh, M. (Eds), Incentives and Performance, Springer, doi: 10.1007/978-3-319-09785-5_8.
Jack, A. (2021), “Business school rankings: the financial times’ experience and evolutions”, Business and Society, May, pp. 1-6, doi: 10.1177/00076503211016783.
Jedidi, K., Schmitt, B.H., Ben Sliman, M. and Li, Y. (2021), “R2M index 1.0: assessing the practical relevance of academic marketing articles”, Journal of Marketing, Vol. 85 No. 5, pp. 22-41, doi: 10.1177/00222429211028145.
Jessani, N.S., Valmeekanathan, A., Babcock, C.M. and Ling, B. (2020), “Academic incentives for enhancing faculty engagement with decision-makers—considerations and recommendations from one School of Public Health”, Humanities and Social Science Communications, Vol. 7, 148, doi: 10.1057/s41599-020-00629-1.
Kaplan, R.S. and Norton, D.P. (1996), The Balanced Scorecard: Translating Strategy into Action, Harvard Business School Press, Boston, MA.
Kianifar, H., Sadeghi, R. and Zarifmahmoudi, L. (2014), “Comparison between Impact factor, Eigenfactor metrics, and SCImago journal rank indicator of pediatric neurology journals”, Acta Informatica Medica, Vol. 22 No. 2, pp. 103-106.
Li, W., Aste, T., Caccioli, F. and Livan, G. (2019), “Early coauthorship with top scientists predicts success in academic careers”, Nature Communications, Vol. 10 No. 1, pp. 1-9.
Merigó, J.M., Zurita, G. and Link-Chaparro, S. (2016), “Normalization of the article influence score between categories”, in Proceedings of the World Congress on Engineering, Vol. 1, pp. 403-408.
Mingers, J. and Yang, W. (2017), “Evaluating journal quality: a Review of journal citation indicators and ranking in business and management”, European Journal of Operational Research, Vol. 257 No. 2, pp. 323-337.
Morales, M. and Calderón, L.F. (1999), “Assessing service quality in schools of business: dimensions of service quality in continuing professional education”, Journal of Economics, Finance and Administrative Sciences, Vol. 5 Nos 9-10, pp. 125-140, doi: 10.46631/jefas.1999.n9-10.06.
O'Brien, J.P., Drnevich, P.L., Crook, T.R. and Armstrong, C.E. (2010), “Does business school research add economic value for students?”, Academy of Management Learning and Education, Vol. 15 No. 3, doi: 10.5465/amle.9.4.zqr638.
Olavarrieta, S. and Villena, M. (2014), “Innovation and business research in Latin America: an overview”, Journal of Business Research, Vol. 67 No. 4, pp. 489-497.
Paulus, F.M., Cruz, N. and Krach, S. (2018), “The impact factor fallacy”, Frontiers in Psychology, Vol. 9, 1487,doi: 10.3389/fpsyg.2018.01487.
Peters, K., Thomas, H. and Smith, R.R. (2018), Rethinking the Business Models of Business Schools: A Critical Review and Change Agenda, Emerald Publishing, Bingley.
Pitt-Watson, D. and Quigley, E. (2019), “Business school rankings for the 21st century”, United Nations Global Compact Report, available at: https://grli.org/resources/business-school-rankings-for-the-21st-century-january-2019/.
Riazanova, O. and McNamara, P. (2015), “Socialization and proactive behavior: multilevel exploration of research productivity drivers in the US business schools”, Academy of Management Learning and Education, Vol. 9 No. 4, doi: 10.5465/amle.2015.0084.
Rizkallah, J. and Sin, D.D. (2010), “Integrative approach to quality assessment of medical journals using impact factor, eigenfactor, and article influence scores”, PLoS One, Vol. 5 No. 4, e10204.
Rosvall, M.J. and Bergstrom, T. (2011), “Multilevel compression of random walks on networks reveals a hierarchical organization in large integrated systems”, PLoS One, Vol. 6 No. 4, e118209.
Rousseau R, the STIMULATE 8 Group (2009), “On the relation between the WoS impact factor, the Eigenfactor, the SCImago Journal Rank, the Article Influence Score and the journal h-index”, available from E-LIS archive, ID: 16448.
RRBM co-founders (2020), “A vision for responsible research in business and management: striving for useful and credible knowledge”, Position paper, available at: www.rrbm.network.
Salcedo, N. (2021a), “Editorial: review and roadmap from the last 10 years (2010-2020)”, Journal of Economics, Finance and Administrative Sciences, Vol. 25 No. 51, pp. 2-6, doi: 10.1108/JEFAS-06-2021-271.
Salcedo, N. (2021b), “Editorial: an upcoming 30th anniversary encouraging the papers' publication”, Journal of Economics, Finance and Administrative Sciences, Vol. 25 No. 52, pp. 178-181, doi: 10.1108/JEFAS-11-2021-329.
Salvador-Oliván, J.A. and Agustín-López, C. (2015), “Correlación entre indicadores bibliométricos en revistas de Web of Science y Scopus”, Revista General de Información y Documentación, Vol. 26 No. 51, pp. 2-6.
Teece, D.J. (2007), “Explicating dynamic capabilities: the nature and microfoundations of (sustainable) enterprise performance”, Strategic Management Journal, Vol. 18 No. 7, pp. 509-533, doi: 10.1002/smj.640.
Teece, D.J. (2017), “Dynamic capabilities and (digital) platform lifecycles”, In Furman, J., Gawer, A., Silverman, B. S. and Stern, S. (Eds.), Advances in Strategic Management, Vol. 37, pp. 211-225, published online, doi: 10.1108/s0742-332220170000037008.
Walters, W.H. (2014), “Do article influence scores overestimate the citation impact of social science journals in subfields that are related to higher-impact natural science disciplines?”, Journal of Informetrics, Vol. 8, pp. 421-430.
Waltman, L. and Van Eck, N. (2010), “The relation between eigenfactor, audience factor, and influence weight”, Journal of the American Society for Information Science and Technology, Vol. 61 No. 7, pp. 1476-1486.
Weller, K. (2015), “Social media and altmetrics: an overview of current alternative approaches to measuring scholarly impact”, in Welpe, I., Wollersheim, J., Ringelhan, S. and Osterloh, M. (Eds), Incentives and Performance, Springer, Cham, doi: 10.1007/978-3-319-09785-5_16.
Welpe, I., Wollersheim, J., Ringelhan, S. and Osterloh, M. (Eds), (2015), Incentives and Performance, Springer, Cham, ebook doi: 10.1007/978-3-319-09785-5.
Zitt, M. and Small, H. (2008), “Modifying the journal impact factor by fractional citation weighting: the audience factor”, Journal of the American Society for Information Science and Technology, Vol. 59 No. 11, pp. 1856-1860.
