{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2026,6,3]],"date-time":"2026-06-03T03:08:20Z","timestamp":1780456100920,"version":"3.54.1"},"reference-count":105,"publisher":"MDPI AG","issue":"7","license":[{"start":{"date-parts":[[2019,6,26]],"date-time":"2019-06-26T00:00:00Z","timestamp":1561507200000},"content-version":"vor","delay-in-days":0,"URL":"https:\/\/creativecommons.org\/licenses\/by\/4.0\/"}],"funder":[{"DOI":"10.13039\/501100001659","name":"Deutsche Forschungsgemeinschaft","doi-asserted-by":"publisher","award":["1736"],"award-info":[{"award-number":["1736"]}],"id":[{"id":"10.13039\/501100001659","id-type":"DOI","asserted-by":"publisher"}]}],"content-domain":{"domain":[],"crossmark-restriction":false},"short-container-title":["Algorithms"],"abstract":"<jats:p>The field of network science is a highly interdisciplinary area; for the empirical analysis of network data, it draws algorithmic methodologies from several research fields. Hence, research procedures and descriptions of the technical results often differ, sometimes widely. In this paper we focus on methodologies for the experimental part of algorithm engineering for network analysis\u2014an important ingredient for a research area with empirical focus. More precisely, we unify and adapt existing recommendations from different fields and propose universal guidelines\u2014including statistical analyses\u2014for the systematic evaluation of network analysis algorithms. This way, the behavior of newly proposed algorithms can be properly assessed and comparisons to existing solutions become meaningful. Moreover, as the main technical contribution, we provide SimexPal, a highly automated tool to perform and analyze experiments following our guidelines. To illustrate the merits of SimexPal and our guidelines, we apply them in a case study: we design, perform, visualize and evaluate experiments of a recent algorithm for approximating betweenness centrality, an important problem in network analysis. In summary, both our guidelines and SimexPal shall modernize and complement previous efforts in experimental algorithmics; they are not only useful for network analysis, but also in related contexts.<\/jats:p>","DOI":"10.3390\/a12070127","type":"journal-article","created":{"date-parts":[[2019,6,27]],"date-time":"2019-06-27T02:43:41Z","timestamp":1561603421000},"page":"127","update-policy":"https:\/\/doi.org\/10.3390\/mdpi_crossmark_policy","source":"Crossref","is-referenced-by-count":19,"title":["Guidelines for Experimental Algorithmics: A Case Study in Network Analysis"],"prefix":"10.3390","volume":"12","author":[{"given":"Eugenio","family":"Angriman","sequence":"first","affiliation":[{"name":"Department of Computer Science, Humboldt-Universit\u00e4t zu Berlin, 10099 Berlin, Germany"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0000-0002-9709-9478","authenticated-orcid":false,"given":"Alexander","family":"van der Grinten","sequence":"additional","affiliation":[{"name":"Department of Computer Science, Humboldt-Universit\u00e4t zu Berlin, 10099 Berlin, Germany"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Moritz","family":"von Looz","sequence":"additional","affiliation":[{"name":"Department of Computer Science, Humboldt-Universit\u00e4t zu Berlin, 10099 Berlin, Germany"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0000-0002-7769-726X","authenticated-orcid":false,"given":"Henning","family":"Meyerhenke","sequence":"additional","affiliation":[{"name":"Department of Computer Science, Humboldt-Universit\u00e4t zu Berlin, 10099 Berlin, Germany"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0000-0003-0454-3937","authenticated-orcid":false,"given":"Martin","family":"N\u00f6llenburg","sequence":"additional","affiliation":[{"name":"Algorithms and Complexity Group, Institute of Logic and Computation, TU Wien, 1040 Vienna, Austria"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0000-0002-9545-0623","authenticated-orcid":false,"given":"Maria","family":"Predari","sequence":"additional","affiliation":[{"name":"Department of Computer Science, Humboldt-Universit\u00e4t zu Berlin, 10099 Berlin, Germany"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Charilaos","family":"Tzovas","sequence":"additional","affiliation":[{"name":"Department of Computer Science, Humboldt-Universit\u00e4t zu Berlin, 10099 Berlin, Germany"}],"role":[{"vocabulary":"crossref","role":"author"}]}],"member":"1968","published-online":{"date-parts":[[2019,6,26]]},"reference":[{"key":"ref_1","doi-asserted-by":"crossref","first-page":"1","DOI":"10.1561\/0400000004","article-title":"Average-Case Complexity","volume":"2","author":"Bogdanov","year":"2006","journal-title":"Found. Trends\u00ae Theor. Comput. Sci."},{"key":"ref_2","doi-asserted-by":"crossref","unstructured":"Spielman, D., and Teng, S.H. (2001, January 6\u20138). Smoothed Analysis of Algorithms: Why the Simplex Algorithm Usually Takes Polynomial Time. Proceedings of the Thirty-Third Annual ACM Symposium on Theory of Computing (STOC\u201901), Crete, Greece.","DOI":"10.1145\/380752.380813"},{"key":"ref_3","doi-asserted-by":"crossref","first-page":"88","DOI":"10.1145\/3232535","article-title":"Beyond worst-case analysis","volume":"62","author":"Roughgarden","year":"2019","journal-title":"Commun. ACM"},{"key":"ref_4","doi-asserted-by":"crossref","unstructured":"Heule, M.J.H., J\u00e4rvisalo, M.J., and Suda, M. (2018). Proceedings of SAT Competition 2018: Solver and Benchmark Descriptions, University of Helsinki.","DOI":"10.3233\/SAT190120"},{"key":"ref_5","unstructured":"Applegate, D.L., Bixby, R.E., Chvatal, V., and Cook, W.J. (2007). The Traveling Salesman Problem: A Computational Study, Princeton University Press."},{"key":"ref_6","unstructured":"Mehlhorn, K., and Sanders, P. (2008). Algorithms and Data Structures: The Basic Toolbox, Springer."},{"key":"ref_7","doi-asserted-by":"crossref","unstructured":"Puglisi, S.J., Smyth, W.F., and Turpin, A.H. (2007). A Taxonomy of Suffix Array Construction Algorithms. ACM Comput. Surv., 39.","DOI":"10.1145\/1242471.1242472"},{"key":"ref_8","unstructured":"Johnson, D.S. (1999). A theoretician\u2019s guide to the experimental analysis of algorithms. DIMACS Workshop Data Structures, Near Neighbor Searches, and Methodology: Fifth and Sixth DIMACS Implementation Challenges, American Mathematical Society."},{"key":"ref_9","doi-asserted-by":"crossref","unstructured":"Muller-Hannemann, M., and Schirra, S. (2010). Algorithm Engineering: Bridging the Gap between Algorithm Theory and Practice, Springer.","DOI":"10.1007\/978-3-642-14866-8"},{"key":"ref_10","doi-asserted-by":"crossref","unstructured":"Moret, B. (2002). Towards a discipline of experimental algorithmics. DIMACS Series in Discrete Mathematics and Theoretical Computer Science, American Mathematical Society.","DOI":"10.1090\/dimacs\/059\/10"},{"key":"ref_11","doi-asserted-by":"crossref","unstructured":"Sanders, P. (2010, January 16). Algorithm Engineering\u2014An Attempt at a Definition Using Sorting as an Example. Proceedings of the 12th Workshop on Algorithm Engineering and Experiments (ALENEX), Austin, TX, USA.","DOI":"10.1137\/1.9781611972900.6"},{"key":"ref_12","doi-asserted-by":"crossref","unstructured":"Bast, H., Delling, D., Goldberg, A., M\u00fcller-Hannemann, M., Pajor, T., Sanders, P., Wagner, D., and Werneck, R.F. (2016). Route planning in transportation networks. Algorithm Engineering, Springer.","DOI":"10.1007\/978-3-319-49487-6_2"},{"key":"ref_13","doi-asserted-by":"crossref","first-page":"1","DOI":"10.1017\/nws.2013.2","article-title":"What is network science?","volume":"1","author":"Brandes","year":"2013","journal-title":"Netw. Sci."},{"key":"ref_14","doi-asserted-by":"crossref","unstructured":"Newman, M. (2018). Networks, Oxford University Press.","DOI":"10.1093\/oso\/9780198805090.001.0001"},{"key":"ref_15","doi-asserted-by":"crossref","first-page":"24","DOI":"10.1287\/ijoc.12.1.24.11899","article-title":"Statistical Analysis of Computational Tests of Algorithms and Heuristics","volume":"12","author":"Coffin","year":"2000","journal-title":"INFORMS J. Comput."},{"key":"ref_16","first-page":"20:1","article-title":"KADABRA is an ADaptive Algorithm for Betweenness via Random Approximation","volume":"57","author":"Borassi","year":"2016","journal-title":"LIPIcs-Leibniz Int. Proc. Inf."},{"key":"ref_17","doi-asserted-by":"crossref","first-page":"508","DOI":"10.1017\/nws.2016.20","article-title":"NetworKit: A tool suite for large-scale complex network analysis","volume":"4","author":"Staudt","year":"2016","journal-title":"Netw. Sci."},{"key":"ref_18","doi-asserted-by":"crossref","first-page":"222","DOI":"10.1080\/15427951.2013.865686","article-title":"Axioms for centrality","volume":"10","author":"Boldi","year":"2014","journal-title":"Internet Math."},{"key":"ref_19","doi-asserted-by":"crossref","first-page":"35","DOI":"10.2307\/3033543","article-title":"A set of measures of centrality based on betweenness","volume":"40","author":"Freeman","year":"1977","journal-title":"Sociometry"},{"key":"ref_20","doi-asserted-by":"crossref","first-page":"163","DOI":"10.1080\/0022250X.2001.9990249","article-title":"A faster algorithm for betweenness centrality","volume":"25","author":"Brandes","year":"2001","journal-title":"J. Math. Sociol."},{"key":"ref_21","doi-asserted-by":"crossref","unstructured":"Bader, D.A., Kintali, S., Madduri, K., and Mihail, M. (2007). Approximating betweenness centrality. International Workshop on Algorithms and Models for the Web-Graph, Springer.","DOI":"10.1007\/978-3-540-77004-6_10"},{"key":"ref_22","unstructured":"Geisberger, R., Sanders, P., and Schultes, D. (June, January 30). Better Approximation of Betweenness Centrality. Proceedings of the Meeting on Algorithm Engineering & Expermiments, Provincetown, MA, USA."},{"key":"ref_23","doi-asserted-by":"crossref","first-page":"438","DOI":"10.1007\/s10618-015-0423-0","article-title":"Fast approximation of betweenness centrality through sampling","volume":"30","author":"Riondato","year":"2016","journal-title":"Data Min. Knowl. Discov."},{"key":"ref_24","doi-asserted-by":"crossref","first-page":"61","DOI":"10.1145\/3208351","article-title":"ABRA: Approximating betweenness centrality in static and dynamic graphs with rademacher averages","volume":"12","author":"Riondato","year":"2018","journal-title":"ACM Trans. Knowl. Discov. Data"},{"key":"ref_25","doi-asserted-by":"crossref","unstructured":"Kunegis, J. (2013, January 13\u201317). Konect: The koblenz network collection. Proceedings of the 22nd International Conference on World Wide Web, Rio de Janeiro, Brazi.","DOI":"10.1145\/2487788.2488173"},{"key":"ref_26","unstructured":"Leskovec, J., and Krevl, A. (2019, March 10). SNAP Datasets: Stanford Large Network Dataset Collection. Available online: http:\/\/snap.stanford.edu\/data."},{"key":"ref_27","unstructured":"Bader, D., Meyerhenke, H., Sanders, P., and Wagner, D. (2012, January 13\u201314). Contemporary Mathematics. Proceedings of the 10th DIMACS Implementation Challenge, Atlanta, GA, USA."},{"key":"ref_28","first-page":"1","article-title":"The University of Florida sparse matrix collection","volume":"38","author":"Davis","year":"2011","journal-title":"ACM Trans. Math. Softw."},{"key":"ref_29","doi-asserted-by":"crossref","unstructured":"Boldi, P., and Vigna, S. (2004, January 17\u201322). The WebGraph Framework I: Compression Techniques. Proceedings of the Thirteenth International World Wide Web Conference (WWW 2004), New York, NY, USA.","DOI":"10.1145\/988672.988752"},{"key":"ref_30","doi-asserted-by":"crossref","first-page":"37","DOI":"10.1145\/2897350.2897355","article-title":"An Interactive Data Repository with Visual Analytics","volume":"17","author":"Rossi","year":"2016","journal-title":"SIGKDD Explor. Newsl."},{"key":"ref_31","doi-asserted-by":"crossref","first-page":"129","DOI":"10.1561\/2200000005","article-title":"A survey of statistical network models","volume":"2","author":"Goldenberg","year":"2010","journal-title":"Found. Trends\u00ae Mach. Learn."},{"key":"ref_32","doi-asserted-by":"crossref","first-page":"046110","DOI":"10.1103\/PhysRevE.78.046110","article-title":"Benchmark graphs for testing community detection algorithms","volume":"78","author":"Lancichinetti","year":"2008","journal-title":"Phys. Rev. E"},{"key":"ref_33","unstructured":"McGeoch, C.C., Sanders, P., Fleischer, R., Cohen, P.R., and Precup, D. (2000). Using Finite Experiments to Study Asymptotic Performance. Experimental Algorithmics, Springer."},{"key":"ref_34","unstructured":"Bernardo, J.M., and Smith, A.F. (2009). Bayesian Theory, John Wiley & Sons."},{"key":"ref_35","doi-asserted-by":"crossref","first-page":"903","DOI":"10.1136\/bmj.331.7521.903","article-title":"Standard deviations and standard errors","volume":"331","author":"Altman","year":"2005","journal-title":"BMJ"},{"key":"ref_36","doi-asserted-by":"crossref","unstructured":"Ellis, P.D. (2010). The Essential Guide to Effect Sizes: Statistical Power, Meta-Analysis, and the Interpretation of Research Results, Cambridge University Press.","DOI":"10.1017\/CBO9780511761676"},{"key":"ref_37","unstructured":"Cohen, J. (1988). Statistical Power Analysis for the Behavioral Sciences, Routledge."},{"key":"ref_38","doi-asserted-by":"crossref","unstructured":"McGeoch, C.C. (2012). A Guide to Experimental Algorithmics, Cambridge University Press. [1st ed.].","DOI":"10.1017\/CBO9780511843747"},{"key":"ref_39","doi-asserted-by":"crossref","first-page":"261","DOI":"10.1023\/A:1011319115230","article-title":"Experimental Evaluation of Heuristic Optimization Algorithms: A Tutorial","volume":"7","author":"Rardin","year":"2001","journal-title":"J. Heuristics"},{"key":"ref_40","doi-asserted-by":"crossref","first-page":"847","DOI":"10.1145\/359619.359631","article-title":"Implementing quicksort programs","volume":"21","author":"Sedgewick","year":"1978","journal-title":"Commun. ACM"},{"key":"ref_41","unstructured":"Eppstein, D., and Wang, J. (2001, January 7\u20139). Fast Approximation of Centrality. Proceedings of the Twelfth Annual ACM-SIAM Symposium on Discrete Algorithms (SODA\u201901), Washington, DC, USA."},{"key":"ref_42","doi-asserted-by":"crossref","unstructured":"James, G., Witten, D., Hastie, T., and Tibshirani, R. (2013). An Introduction to Statistical Learning\u2014With Applications in R, Springer.","DOI":"10.1007\/978-1-4614-7138-7"},{"key":"ref_43","doi-asserted-by":"crossref","unstructured":"Hinton, G.E. (2012). A practical guide to training restricted Boltzmann machines. Neural Networks: Tricks of the Trade, Springer.","DOI":"10.1007\/978-3-642-35289-8_32"},{"key":"ref_44","doi-asserted-by":"crossref","unstructured":"Larochelle, H., Erhan, D., Courville, A., Bergstra, J., and Bengio, Y. (2007, January 20\u201324). An empirical evaluation of deep architectures on problems with many factors of variation. Proceedings of the 24th International Conference on Machine Learning, Corvalis, OR, USA.","DOI":"10.1145\/1273496.1273556"},{"key":"ref_45","doi-asserted-by":"crossref","first-page":"2278","DOI":"10.1109\/5.726791","article-title":"Gradient-based learning applied to document recognition","volume":"86","author":"LeCun","year":"1998","journal-title":"Proc. IEEE"},{"key":"ref_46","unstructured":"Bergamini, E., Meyerhenke, H., and Staudt, C.L. (2015, January 5). Approximating Betweenness Centrality in Large Evolving Networks. Proceedings of the Seventeenth Workshop on Algorithm Engineering and Experiments (ALENEX), San Diego, CA, USA."},{"key":"ref_47","doi-asserted-by":"crossref","unstructured":"Green, O., McColl, R., and Bader, D.A. (2012, January 3\u20135). A Fast Algorithm for Streaming Betweenness Centrality. Proceedings of the 2012 ASE\/IEEE International Conference on Social Computing and 2012 ASE\/IEEE International Conference on Privacy, Security, Risk and Trust, Amsterdam, The Netherlands.","DOI":"10.1109\/SocialCom-PASSAT.2012.37"},{"key":"ref_48","doi-asserted-by":"crossref","first-page":"2494","DOI":"10.1109\/TKDE.2015.2419666","article-title":"Scalable Online Betweenness Centrality in Evolving Graphs","volume":"27","author":"Kourtellis","year":"2015","journal-title":"IEEE Trans. Knowl. Data Eng."},{"key":"ref_49","doi-asserted-by":"crossref","first-page":"2303","DOI":"10.1142\/S0218127407018403","article-title":"Centrality estimation in large networks","volume":"17","author":"Brandes","year":"2007","journal-title":"Int. J. Bifurc. Chaos"},{"key":"ref_50","unstructured":"Bader, D., and Madduri, K. (2006, January 14\u201318). Parallel Algorithms for Evaluating Centrality Indices in Real-world Networks. Proceedings of the 2006 International Conference on Parallel Processing (ICPP\u201906), Columbus, OH, USA."},{"key":"ref_51","doi-asserted-by":"crossref","unstructured":"Rizi, F.S., Schloetterer, J., and Granitzer, M. (2018, January 28\u201331). Shortest Path Distance Approximation Using Deep Learning Techniques. Proceedings of the IEEE\/ACM International Conference on Advances in Social Networks Analysis and Mining (ASONAM), Barcelona, Spain.","DOI":"10.1109\/ASONAM.2018.8508763"},{"key":"ref_52","doi-asserted-by":"crossref","unstructured":"Akiba, T., Iwata, Y., and Yoshida, Y. (2013, January 22\u201327). Fast exact shortest-path distance queries on large networks by pruned landmark labeling. Proceedings of the 2013 ACM SIGMOD International Conference on Management of Data, New York, NY, USA.","DOI":"10.1145\/2463676.2465315"},{"key":"ref_53","unstructured":"Penschuck, M. (2017, January 21\u201323). Generating practical random hyperbolic graphs in near-linear time and with sub-linear memory. Proceedings of the 16th International Symposium on Experimental Algorithms (SEA 2017), London, UK."},{"key":"ref_54","doi-asserted-by":"crossref","unstructured":"McLaughlin, A., and Bader, D.A. (2014, January 16\u201321). Scalable and High Performance Betweenness Centrality on the GPU. Proceedings of the International Conference for High Performance Computing, Networking, Storage and Analysis, New Orleans, LA, USA.","DOI":"10.1109\/SC.2014.52"},{"key":"ref_55","doi-asserted-by":"crossref","unstructured":"Sariy\u00fcce, A.E., Kaya, K., Saule, E., and \u00c7ataly\u00fcrek, U.V. (2013, January 16). Betweenness Centrality on GPUs and Heterogeneous Architectures. Proceedings of the 6th Workshop on General Purpose Processor Using Graphics Processing Units, Houston, TX, USA.","DOI":"10.1145\/2458523.2458531"},{"key":"ref_56","doi-asserted-by":"crossref","unstructured":"Shi, Z., and Zhang, B. (2011). Fast network centrality analysis using GPUs. BMC Bioinform., 12.","DOI":"10.1186\/1471-2105-12-149"},{"key":"ref_57","doi-asserted-by":"crossref","unstructured":"Bampis, E. (2015). Greedily Improving Our Own Centrality in A Network. Experimental Algorithms, Springer.","DOI":"10.1007\/978-3-319-20086-6"},{"key":"ref_58","unstructured":"Hennessy, J.L., and Patterson, D.A. (2011). Computer Architecture: A Quantitative Approach, Elsevier."},{"key":"ref_59","doi-asserted-by":"crossref","first-page":"3","DOI":"10.1287\/opre.50.1.3.17780","article-title":"Solving real-world linear programs: A decade and more of progress","volume":"50","author":"Bixby","year":"2002","journal-title":"Oper. Res."},{"key":"ref_60","doi-asserted-by":"crossref","first-page":"142","DOI":"10.1017\/S0025557200174534","article-title":"88.27 more on spreads and non-arithmetic means","volume":"88","author":"Mitchell","year":"2004","journal-title":"Math. Gaz."},{"key":"ref_61","doi-asserted-by":"crossref","first-page":"1202","DOI":"10.1145\/63039.63043","article-title":"Characterizing Computer Performance with a Single Number","volume":"31","author":"Smith","year":"1988","journal-title":"Commun. ACM"},{"key":"ref_62","doi-asserted-by":"crossref","first-page":"218","DOI":"10.1145\/5666.5673","article-title":"How Not To Lie With Statistics: The Correct Way To Summarize Benchmark Results","volume":"29","author":"Fleming","year":"1986","journal-title":"Commun. ACM"},{"key":"ref_63","doi-asserted-by":"crossref","unstructured":"AlGhamdi, Z., Jamour, F., Skiadopoulos, S., and Kalnis, P. (2017, January 27\u201329). A Benchmark for Betweenness Centrality Approximation Algorithms on Large Graphs. Proceedings of the 29th International Conference on Scientific and Statistical Database Management, Chicago, IL, USA.","DOI":"10.1145\/3085504.3085510"},{"key":"ref_64","doi-asserted-by":"crossref","first-page":"1138","DOI":"10.1287\/opre.18.6.1138","article-title":"The Traveling-Salesman Problem and Minimum Spanning Trees","volume":"18","author":"Held","year":"1970","journal-title":"Oper. Res."},{"key":"ref_65","unstructured":"(2019, March 21). Git. Available online: https:\/\/git-scm.com\/."},{"key":"ref_66","unstructured":"(2019, June 30). GitHub. Available online: https:\/\/github.com\/."},{"key":"ref_67","unstructured":"(2019, May 12). GitLab. Available online: https:\/\/gitlab.com\/."},{"key":"ref_68","unstructured":"(2019, May 12). Bitbucket. Available online: https:\/\/bitbucket.org\/."},{"key":"ref_69","doi-asserted-by":"crossref","first-page":"89","DOI":"10.1145\/1273442.1250746","article-title":"Valgrind: A framework for heavyweight dynamic binary instrumentation","volume":"42","author":"Nethercote","year":"2007","journal-title":"ACM Sigplan Not."},{"key":"ref_70","unstructured":"Reinders, J. (2005). VTune Performance Analyzer Essentials, Intel Press."},{"key":"ref_71","doi-asserted-by":"crossref","unstructured":"Hamadi, Y., and Wintersteiger, C.M. (2012, January 22\u201326). Seven Challenges in Parallel SAT Solving. Proceedings of the Twenty-Sixth AAAI Conference on Artificial Intelligence, Toronto, ON, Canada.","DOI":"10.1609\/aaai.v26i1.8438"},{"key":"ref_72","unstructured":"Kimmig, R., Meyerhenke, H., and Strash, D. (June, January 29). Shared Memory Parallel Subgraph Enumeration. Proceedings of the IEEE International Parallel and Distributed Processing Symposium Workshops, Orlando, FL, USA."},{"key":"ref_73","doi-asserted-by":"crossref","first-page":"5","DOI":"10.1145\/103162.103163","article-title":"What Every Computer Scientist Should Know About Floating-point Arithmetic","volume":"23","author":"Goldberg","year":"1991","journal-title":"ACM Comput. Surv."},{"key":"ref_74","unstructured":"YAML (2019, March 22). The YAML Project. Available online: https:\/\/yaml.org\/."},{"key":"ref_75","unstructured":"German Research Foundation (DFG) (2015). DFG Guidelines on the Handling of Research Data, German Research Foundation."},{"key":"ref_76","unstructured":"(2019, June 08). JSON. Available online: https:\/\/www.json.org\/index.html."},{"key":"ref_77","doi-asserted-by":"crossref","first-page":"17","DOI":"10.1080\/00031305.1973.10478966","article-title":"Graphs in Statistical Analysis","volume":"27","author":"Anscombe","year":"1973","journal-title":"Am Stat."},{"key":"ref_78","unstructured":"Healy, K. (2018). Data Visualization: A Practical Introduction, Princeton University Press."},{"key":"ref_79","doi-asserted-by":"crossref","first-page":"181","DOI":"10.1007\/3-540-36383-1_9","article-title":"Presenting Data from Experiments in Algorithmics","volume":"Volume 2547","author":"Fleischer","year":"2002","journal-title":"Experimental Algorithmics"},{"key":"ref_80","unstructured":"Tufte, E.R. (2001). The Visual Display of Quantitative Information, Graphics Press."},{"key":"ref_81","doi-asserted-by":"crossref","unstructured":"Munzner, T. (2014). Visualization Analyis and Design, CRC Press.","DOI":"10.1201\/b17511"},{"key":"ref_82","unstructured":"Ware, C. (2012). Information Visualization: Perception for Design, Morgan Kaufmann. [3rd ed.]."},{"key":"ref_83","doi-asserted-by":"crossref","first-page":"912","DOI":"10.2307\/3803199","article-title":"Null hypothesis testing: Problems, prevalence, and an alternative","volume":"64","author":"Anderson","year":"2000","journal-title":"J. Wildl. Manag."},{"key":"ref_84","doi-asserted-by":"crossref","first-page":"1","DOI":"10.1080\/01973533.2015.1012991","article-title":"Editorial","volume":"37","author":"Trafimow","year":"2015","journal-title":"Basic Appl. Soc. Psychol."},{"key":"ref_85","doi-asserted-by":"crossref","first-page":"129","DOI":"10.1080\/00031305.2016.1154108","article-title":"The ASA\u2019s statement on p-values: Context, process, and purpose","volume":"70","author":"Wasserstein","year":"2016","journal-title":"Am. Stat."},{"key":"ref_86","doi-asserted-by":"crossref","first-page":"627","DOI":"10.1093\/aje\/kwx261","article-title":"The harm done to reproducibility by the culture of null hypothesis significance testing","volume":"186","author":"Lash","year":"2017","journal-title":"Am. J. Epidemiol."},{"key":"ref_87","doi-asserted-by":"crossref","first-page":"7","DOI":"10.1177\/0956797613504966","article-title":"The New Statistics: Why and How","volume":"25","author":"Cumming","year":"2014","journal-title":"Psychol. Sci."},{"key":"ref_88","doi-asserted-by":"crossref","first-page":"178","DOI":"10.3758\/s13423-016-1221-4","article-title":"The Bayesian New Statistics: Hypothesis testing, estimation, meta-analysis, and power analysis from a Bayesian perspective","volume":"25","author":"Kruschke","year":"2018","journal-title":"Psychon. Bull. Rev."},{"key":"ref_89","doi-asserted-by":"crossref","first-page":"611","DOI":"10.1890\/13-0590.1","article-title":"In defense of P values","volume":"95","author":"Murtaugh","year":"2014","journal-title":"Ecology"},{"key":"ref_90","doi-asserted-by":"crossref","first-page":"791","DOI":"10.1080\/01621459.1976.10480949","article-title":"Science and Statistics","volume":"71","author":"Box","year":"1976","journal-title":"J. Am. Stat. Assoc."},{"key":"ref_91","doi-asserted-by":"crossref","first-page":"171","DOI":"10.1007\/BF01206525","article-title":"\u00dcber den zentralen Grenzwertsatz der Wahrscheinlichkeitsrechnung und das Momentenproblem","volume":"8","year":"1920","journal-title":"Math. Z."},{"key":"ref_92","doi-asserted-by":"crossref","first-page":"169","DOI":"10.1080\/01621459.1976.10481508","article-title":"The Equivalence of Generalized Least Squares and Maximum Likelihood Estimates in the Exponential Family","volume":"71","author":"Charnes","year":"1976","journal-title":"J. Am. Stat. Assoc."},{"key":"ref_93","doi-asserted-by":"crossref","unstructured":"Fisher, R.A. (1992). Statistical methods for research workers. Breakthroughs in Statistics, Springer.","DOI":"10.1007\/978-1-4612-4380-9_6"},{"key":"ref_94","doi-asserted-by":"crossref","unstructured":"Young, G.A., and Smith, R.L. (2005). Essentials of Statistical Inference, Cambridge University Press.","DOI":"10.1017\/CBO9780511755392"},{"key":"ref_95","doi-asserted-by":"crossref","first-page":"80","DOI":"10.2307\/3001968","article-title":"Individual comparisons by ranking methods","volume":"1","author":"Wilcoxon","year":"1945","journal-title":"Biomed. Bull."},{"key":"ref_96","unstructured":"Jones, E., Oliphant, T., and Peterson, P. (2019, June 14). SciPy: Open Source Scientific Tools for Python. Available online: https:\/\/www.scipy.org."},{"key":"ref_97","doi-asserted-by":"crossref","first-page":"52","DOI":"10.1080\/01621459.1961.10482090","article-title":"Multiple comparisons among means","volume":"56","author":"Dunn","year":"1961","journal-title":"J. Am. Stat. Assoc."},{"key":"ref_98","first-page":"65","article-title":"A Simple Sequentially Rejective Multiple Test Procedure","volume":"6","author":"Holm","year":"1979","journal-title":"Scand. J. Stat."},{"key":"ref_99","doi-asserted-by":"crossref","unstructured":"Smithson, M. (2003). Confidence Intervals, SAGE Publications, Inc.","DOI":"10.4135\/9781412983761"},{"key":"ref_100","doi-asserted-by":"crossref","first-page":"333","DOI":"10.1098\/rsta.1937.0005","article-title":"Outline of a Theory of Statistical Estimation Based on the Classical Theory of Probability","volume":"236","author":"Neyman","year":"1937","journal-title":"Philos. Trans. R. Soc. Lond. A"},{"key":"ref_101","doi-asserted-by":"crossref","unstructured":"Gelman, A., Stern, H.S., Carlin, J.B., Dunson, D.B., Vehtari, A., and Rubin, D.B. (2013). Bayesian Data Analysis, Chapman Press.","DOI":"10.1201\/b16018"},{"key":"ref_102","doi-asserted-by":"crossref","first-page":"e55","DOI":"10.7717\/peerj-cs.55","article-title":"Probabilistic programming in Python using PyMC3","volume":"2","author":"Salvatier","year":"2016","journal-title":"PeerJ Comput. Sci."},{"key":"ref_103","doi-asserted-by":"crossref","unstructured":"Jeffreys, H. (1998). The Theory of Probability, Oxford University Press.","DOI":"10.1093\/oso\/9780198503682.001.0001"},{"key":"ref_104","doi-asserted-by":"crossref","unstructured":"Von Looz, M., Meyerhenke, H., and Prutkin, R. (2015, January 9\u201311). Generating random hyperbolic graphs in subquadratic time. Proceedings of the International Symposium on Algorithms and Computation, Nagoya, Japan.","DOI":"10.1007\/978-3-662-48971-0_40"},{"key":"ref_105","doi-asserted-by":"crossref","first-page":"716","DOI":"10.1109\/TAC.1974.1100705","article-title":"A new look at the statistical model identification","volume":"19","author":"Akaike","year":"1974","journal-title":"IEEE Trans. Autom. Control"}],"container-title":["Algorithms"],"original-title":[],"language":"en","link":[{"URL":"https:\/\/www.mdpi.com\/1999-4893\/12\/7\/127\/pdf","content-type":"unspecified","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2025,10,11]],"date-time":"2025-10-11T13:01:26Z","timestamp":1760187686000},"score":1,"resource":{"primary":{"URL":"https:\/\/www.mdpi.com\/1999-4893\/12\/7\/127"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2019,6,26]]},"references-count":105,"journal-issue":{"issue":"7","published-online":{"date-parts":[[2019,7]]}},"alternative-id":["a12070127"],"URL":"https:\/\/doi.org\/10.3390\/a12070127","relation":{},"ISSN":["1999-4893"],"issn-type":[{"value":"1999-4893","type":"electronic"}],"subject":[],"published":{"date-parts":[[2019,6,26]]}}}