{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2026,4,18]],"date-time":"2026-04-18T14:27:46Z","timestamp":1776522466886,"version":"3.51.2"},"reference-count":51,"publisher":"MDPI AG","issue":"1","license":[{"start":{"date-parts":[[2019,1,10]],"date-time":"2019-01-10T00:00:00Z","timestamp":1547078400000},"content-version":"vor","delay-in-days":0,"URL":"https:\/\/creativecommons.org\/licenses\/by\/4.0\/"}],"content-domain":{"domain":[],"crossmark-restriction":false},"short-container-title":["Publications"],"abstract":"<jats:p>While output and impact assessments were initially at the forefront of institutional research evaluations, efficiency measurements have become popular in recent years. Research efficiency is measured by indicators that relate research output to input. The additional consideration of research input in research evaluation is obvious, since the output depends on the input. The present study is based on a comprehensive dataset with input and output data for 50 US universities. As input, we used research expenses, and as output the number of highly-cited papers. We employed Data Efficiency Analysis (DEA), Free Disposal Hull (FDH) and two more robust models: the order-m and order-\u03b1 approaches. The results of the DEA and FDH analysis show that Harvard University and Boston College can be called especially efficient compared to the other universities. While the strength of Harvard University lies in its high output of highly-cited papers, the strength of Boston College is its small input. In the order-\u03b1 and order-m frameworks, Harvard University remains efficient, but Boston College becomes super-efficient. We produced university rankings based on adjusted efficiency scores (subsequent to regression analyses), in which single covariates (e.g., the disciplinary profile) are held constant.<\/jats:p>","DOI":"10.3390\/publications7010004","type":"journal-article","created":{"date-parts":[[2019,1,11]],"date-time":"2019-01-11T04:10:16Z","timestamp":1547179816000},"page":"4","update-policy":"https:\/\/doi.org\/10.3390\/mdpi_crossmark_policy","source":"Crossref","is-referenced-by-count":13,"title":["How Efficiently Do Elite US Universities Produce Highly Cited Papers?"],"prefix":"10.3390","volume":"7","author":[{"ORCID":"https:\/\/orcid.org\/0000-0002-5651-2641","authenticated-orcid":false,"given":"Klaus","family":"Wohlrabe","sequence":"first","affiliation":[{"name":"Ifo Institute, Poschingerstr. 5, 81679 Munich, Germany"}]},{"ORCID":"https:\/\/orcid.org\/0000-0002-0255-8628","authenticated-orcid":false,"given":"F\u00e9lix","family":"de Moya Anegon","sequence":"additional","affiliation":[{"name":"CSIC, Institute of Public Goods and Policies (IPP) Consejo Superior de Investigaciones Cient\u00edficas C\/Albasanz, 26-28 28037 Madrid, Spain"}]},{"ORCID":"https:\/\/orcid.org\/0000-0003-0810-7091","authenticated-orcid":false,"given":"Lutz","family":"Bornmann","sequence":"additional","affiliation":[{"name":"Division for Science and Innovation Studies, Administrative Headquarters of the Max Planck Society Hofgartenstr. 8, 80539 Munich, Germany"}]}],"member":"1968","published-online":{"date-parts":[[2019,1,10]]},"reference":[{"key":"ref_1","doi-asserted-by":"crossref","unstructured":"Ziman, J. (2000). Real Science. What It Is, and What It Means, Cambridge University Press.","DOI":"10.1017\/CBO9780511541391"},{"key":"ref_2","doi-asserted-by":"crossref","first-page":"199","DOI":"10.1002\/aris.2011.1440450112","article-title":"Scientific peer review","volume":"45","author":"Bornmann","year":"2011","journal-title":"Annu. Rev. Inf. Sci. Technol."},{"key":"ref_3","doi-asserted-by":"crossref","unstructured":"Wilsdon, J., Allen, L., Belfiore, E., Campbell, P., Curry, S., Hill, S., Jones, R., Kain, R., Kerridge, S., and Thelwall, M. (2015). The Metric Tide: Report of the Independent Review of the Role of Metrics in Research Assessment and Management, Higher Education Funding Council for England (HEFCE).","DOI":"10.4135\/9781473978782"},{"key":"ref_4","unstructured":"National Research Council (2014). Furthering America\u2019s Research Enterprise, The National Academies Press."},{"key":"ref_5","first-page":"567","article-title":"Evaluation of Scientific Production in Mexican State Public Universities (2007\u20132011) Using Principal Component Analysis","volume":"24","author":"Tarango","year":"2015","journal-title":"Prof. Inf."},{"key":"ref_6","doi-asserted-by":"crossref","unstructured":"Hazelkorn, E. (2011). Rankings and the Reshaping of Higher Education. The Battle for World-Class Excellence, Palgrave Macmillan.","DOI":"10.1057\/9780230306394"},{"key":"ref_7","doi-asserted-by":"crossref","unstructured":"Welpe, I.M., Wollersheim, J., Ringelhan, S., and Osterloh, M. (2015). The Informative Value of International University Rankings: Some Methodological Remarks. Incentives and Performance, Springer International Publishing.","DOI":"10.1007\/978-3-319-09785-5"},{"key":"ref_8","doi-asserted-by":"crossref","first-page":"918","DOI":"10.1016\/j.ejor.2015.02.005","article-title":"Rankings and university performance: A conditional multidimensional approach","volume":"244","author":"Daraio","year":"2015","journal-title":"Eur. J. Oper. Res."},{"key":"ref_9","doi-asserted-by":"crossref","first-page":"581","DOI":"10.1016\/j.joi.2014.04.008","article-title":"What is the effect of country-specific characteristics on the research performance of scientific institutions? Using multi-level statistical models to rank and map universities and research-focused institutions worldwide","volume":"8","author":"Bornmann","year":"2014","journal-title":"J. Informetr."},{"key":"ref_10","doi-asserted-by":"crossref","first-page":"223","DOI":"10.1007\/s11192-013-0986-8","article-title":"What do global university rankings really measure? The search for the X factor and the X entity","volume":"97","year":"2013","journal-title":"Scientometrics"},{"key":"ref_11","doi-asserted-by":"crossref","first-page":"1988","DOI":"10.1002\/asi.23314","article-title":"Multidimensional assessment of scholarly research impact","volume":"66","author":"Moed","year":"2015","journal-title":"J. Assoc. Inf. Sci. Technol."},{"key":"ref_12","doi-asserted-by":"crossref","first-page":"646","DOI":"10.1016\/j.joi.2016.04.006","article-title":"A farewell to the MNCS and like size-independent indicators","volume":"10","author":"Abramo","year":"2016","journal-title":"J. Informetr."},{"key":"ref_13","doi-asserted-by":"crossref","first-page":"652","DOI":"10.1016\/j.joi.2015.11.009","article-title":"Efficiency of research performance and the glass researcher","volume":"10","author":"Bornmann","year":"2016","journal-title":"J. Informetr."},{"key":"ref_14","doi-asserted-by":"crossref","first-page":"671","DOI":"10.1016\/j.joi.2015.12.008","article-title":"The elephant in the room: The problem of quantifying productivity in evaluative scientometrics","volume":"10","author":"Waltman","year":"2016","journal-title":"J. Informetr."},{"key":"ref_15","doi-asserted-by":"crossref","unstructured":"Gralka, S., Wohlrabe, K., and Bornmann, L. (2018). How to Measure Research Efficiency in Higher Education? Research Grants vs. Publication Output. J. High. Educ. Policy Manag., in press.","DOI":"10.2139\/ssrn.3210550"},{"key":"ref_16","doi-asserted-by":"crossref","first-page":"1129","DOI":"10.1007\/s11192-014-1269-8","article-title":"How do you define and measure research productivity?","volume":"101","author":"Abramo","year":"2014","journal-title":"Scientometrics"},{"key":"ref_17","doi-asserted-by":"crossref","first-page":"1","DOI":"10.1016\/S0304-4076(01)00080-X","article-title":"Nonparametric frontier estimation: A robust approach","volume":"106","author":"Cazals","year":"2002","journal-title":"J. Econom."},{"key":"ref_18","doi-asserted-by":"crossref","first-page":"358","DOI":"10.1017\/S0266466605050206","article-title":"Nonparametric frontier estimation: A conditional quantile-based approach","volume":"21","author":"Aragon","year":"2005","journal-title":"Econom. Theory"},{"key":"ref_19","doi-asserted-by":"crossref","first-page":"175","DOI":"10.3102\/00346543052002175","article-title":"Institutional Performance in Higher-Education\u2014The Efficiency Dimension","volume":"52","author":"Lindsay","year":"1982","journal-title":"Rev. Educ. Res."},{"key":"ref_20","doi-asserted-by":"crossref","first-page":"82","DOI":"10.1177\/0013161X83019002006","article-title":"Evaluation of Educational-Program Proposals by Means of Dea","volume":"19","author":"Bessent","year":"1983","journal-title":"Educ. Adm. Q."},{"key":"ref_21","doi-asserted-by":"crossref","first-page":"245","DOI":"10.1080\/09645290110086126","article-title":"An Empirical Survey of Frontier Efficiency Measurement Techniques in Education","volume":"9","author":"Worthington","year":"2001","journal-title":"Educ. Econ."},{"key":"ref_22","doi-asserted-by":"crossref","first-page":"581","DOI":"10.1007\/s11192-016-2173-1","article-title":"Measurement and determinants of academic research efficiency: A systematic review of the evidence","volume":"110","author":"Rhaiem","year":"2017","journal-title":"Scientometrics"},{"key":"ref_23","doi-asserted-by":"crossref","first-page":"339","DOI":"10.1057\/jors.2015.92","article-title":"Efficiency in education: A review of literature and a way forward","volume":"68","year":"2017","journal-title":"J. Oper. Res. Soc."},{"key":"ref_24","doi-asserted-by":"crossref","first-page":"389","DOI":"10.1007\/s11192-006-0028-x","article-title":"Advanced indicators of productivity of universities. An application of robust nonparametric methods to Italian data","volume":"66","author":"Bonaccorsi","year":"2006","journal-title":"Scientometrics"},{"key":"ref_25","unstructured":"Bonaccorsi, A., Daraio, C., Raty, T., and Simar, L. (2007). Efficiency and University Size: Discipline-Wise Evidence from European Universities, University Library of Munich. MPRA Paper 10265."},{"key":"ref_26","doi-asserted-by":"crossref","first-page":"1595","DOI":"10.1057\/jors.2012.115","article-title":"Accounting for economies of scope in performance evaluations of university professors","volume":"64","author":"Rogge","year":"2013","journal-title":"J. Oper. Res. Soc."},{"key":"ref_27","unstructured":"Bruffaerts, C., Rock, B.D., and Dehon, C. (2013). The Research Efficiency of US Universities: A Nonparametric Frontier Modelling Approach, Universite Libre de Bruxelles. Working Papers ECARES ECARES 2013-31."},{"key":"ref_28","doi-asserted-by":"crossref","unstructured":"Gnewuch, M., and Wohlrabe, K. (2018). Super-efficiency of education institutions: An application to economics departments. Educ. Econ., in press.","DOI":"10.2139\/ssrn.3198491"},{"key":"ref_29","doi-asserted-by":"crossref","first-page":"284","DOI":"10.2307\/1926974","article-title":"Institutions of Higher-Education as Multi-Product Firms\u2014Economies of Scale and Scope","volume":"71","author":"Cohn","year":"1989","journal-title":"Rev. Econ. Stat."},{"key":"ref_30","doi-asserted-by":"crossref","first-page":"369","DOI":"10.1353\/rhe.2005.0011","article-title":"An examination of costs at four-year public colleges and universities between 1989 and 1998","volume":"28","author":"Harter","year":"2005","journal-title":"Rev. High. Educ."},{"key":"ref_31","doi-asserted-by":"crossref","first-page":"429","DOI":"10.1023\/B:RIHE.0000027394.33115.71","article-title":"Do costs differ between for-profit and not-for-profit producers of higher education?","volume":"45","author":"Laband","year":"2004","journal-title":"Res. High. Educ."},{"key":"ref_32","first-page":"11","article-title":"Stochastic Cost Inefficiency Estimates and Rankings of Public and Private Research and Doctoral Granting Universities","volume":"4","author":"Sav","year":"2012","journal-title":"J. Knowl. Manag. Econ. Inf. Technol."},{"key":"ref_33","doi-asserted-by":"crossref","first-page":"60","DOI":"10.1080\/03075079.2013.818644","article-title":"Efficiency, costs, rankings and heterogeneity: The case of US higher education","volume":"40","author":"Agasisti","year":"2015","journal-title":"Stud. High. Educ."},{"key":"ref_34","doi-asserted-by":"crossref","first-page":"469","DOI":"10.1007\/s11162-016-9434-y","article-title":"Are Public Master\u2019s Institutions Cost Efficient? A Stochastic Frontier and Spatial Analysis","volume":"58","author":"Titus","year":"2017","journal-title":"Res. High. Educ."},{"key":"ref_35","doi-asserted-by":"crossref","first-page":"338","DOI":"10.1016\/0377-2217(79)90229-7","article-title":"Measuring the Efficiency of Decision-Making Units","volume":"3","author":"Charnes","year":"1979","journal-title":"Eur. J. Oper. Res."},{"key":"ref_36","doi-asserted-by":"crossref","unstructured":"Bogetoft, P., and Otto, L. (2011). Benchmarking with DEA, SFA and R, Springer.","DOI":"10.1007\/978-1-4419-7961-2"},{"key":"ref_37","unstructured":"Wilson, P.W., and Clemson, S.C. (2013). FEAR 2.0: A Software Package for Frontier Analysis with R, Department of Economics, Clemson University."},{"key":"ref_38","doi-asserted-by":"crossref","unstructured":"Chander, P., Dr\u00e8ze, J., Lovell, C.K., and Mintz, J. (2006). Measuring Labor-Efficiency in Post Offices. Public Goods, Environmental Externalities and Fiscal Competition, Springer US.","DOI":"10.1007\/b135529"},{"key":"ref_39","unstructured":"Daraio, C., and Simar, L. (2007). Advanced Robust and Nonparametric Methods in Efficiency Analysis: Methodology and Applications, Springer."},{"key":"ref_40","doi-asserted-by":"crossref","first-page":"128","DOI":"10.1093\/scipol\/scv026","article-title":"Exploring efficiency differentials between Italian and Polish universities, 2001\u20132011","volume":"43","author":"Agasisti","year":"2015","journal-title":"Sci. Public Policy"},{"key":"ref_41","unstructured":"StataCorp (2015). Stata Statistical Software: Release 14, Stata Corporation."},{"key":"ref_42","doi-asserted-by":"crossref","first-page":"2310","DOI":"10.1002\/asi.22923","article-title":"Do universities or research institutions with a specific subject profile have an advantage or a disadvantage in institutional rankings? A latent class analysis with data from the SCImago ranking","volume":"64","author":"Bornmann","year":"2013","journal-title":"J. Am. Soc. Inf. Sci. Technol."},{"key":"ref_43","doi-asserted-by":"crossref","unstructured":"Angeles, G., Cronin, C., Guilkey, D.K., Lance, P.M., and Sullivan, B.A. (2014). Guide to Longitudinal Program Impact Evaluation, Measurement, Learning & Evaluation Project.","DOI":"10.2139\/ssrn.2796689"},{"key":"ref_44","doi-asserted-by":"crossref","unstructured":"Bonaccorsi, A. (2014). Knowledge, Diversity and Performance in Europe\u2014An Higher Education: A Changing Landscape, Edvard Elgar.","DOI":"10.4337\/9781783472000"},{"key":"ref_45","unstructured":"Eumida (2009). Final Study Report: Feasibility Study for Creating a European University Data Collection, European Commission, Research Directorate-General C-European Research Area Universities and Researchers."},{"key":"ref_46","doi-asserted-by":"crossref","first-page":"225","DOI":"10.1007\/s11192-007-1942-2","article-title":"The measurement of Italian universities\u2019 research productivity by a non parametric-bibliometric methodology","volume":"76","author":"Abramo","year":"2008","journal-title":"Scientometrics"},{"key":"ref_47","doi-asserted-by":"crossref","first-page":"393","DOI":"10.1023\/B:REIO.0000037538.48594.2c","article-title":"Performance differences in German higher education: Empirical analysis of strategic groups","volume":"24","author":"Warning","year":"2004","journal-title":"Rev. Ind. Organ."},{"key":"ref_48","doi-asserted-by":"crossref","first-page":"37","DOI":"10.1177\/016555159502100104","article-title":"A Bibliometric Study on Aging and Reception Processes of Scientific Literature","volume":"21","year":"1995","journal-title":"J. Inf. Sci."},{"key":"ref_49","unstructured":"Waltman, L., and van Eck, N.J. (2016, January 14\u201316). The need for contextualized scientometric analysis: An opinion paper. Proceedings of the 21st International Conference on Science and Technology Indicator, Valencia, Spain."},{"key":"ref_50","doi-asserted-by":"crossref","first-page":"658","DOI":"10.1016\/j.joi.2016.04.008","article-title":"Productivity, performance, efficiency, impact\u2014What do we measure anyway?: Some comments on the paper \u201cA farewell to the MNCS and like size-independent indicators\u201d by Abramo and D\u2019Angelo","volume":"10","author":"Thijs","year":"2016","journal-title":"J. Informetr."},{"key":"ref_51","doi-asserted-by":"crossref","first-page":"518","DOI":"10.1093\/scipol\/scv058","article-title":"Research funding and national academic performance: Examination of a Danish success story","volume":"43","author":"Aagaard","year":"2016","journal-title":"Sci. Public Policy"}],"container-title":["Publications"],"original-title":[],"language":"en","link":[{"URL":"https:\/\/www.mdpi.com\/2304-6775\/7\/1\/4\/pdf","content-type":"unspecified","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2025,10,11]],"date-time":"2025-10-11T12:24:55Z","timestamp":1760185495000},"score":1,"resource":{"primary":{"URL":"https:\/\/www.mdpi.com\/2304-6775\/7\/1\/4"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2019,1,10]]},"references-count":51,"journal-issue":{"issue":"1","published-online":{"date-parts":[[2019,3]]}},"alternative-id":["publications7010004"],"URL":"https:\/\/doi.org\/10.3390\/publications7010004","relation":{},"ISSN":["2304-6775"],"issn-type":[{"value":"2304-6775","type":"electronic"}],"subject":[],"published":{"date-parts":[[2019,1,10]]}}}